Codepagex

https://github.com/tallakt/codepagex
Admin
Codepagex is an Elixir library for converting between string encodings to and from UTF-8, similar to...
Tokens:5,150
Snippets:53
Trust Score:7.3
License:Apache-2.0
Update:6 months ago
Show doc for...
Tokens:
Raw
### Retrieve encoding aliases

Source: https://context7.com/tallakt/codepagex/llms.txt

Get a map of shorthand aliases to their full encoding names.

```elixir
# Get currently available aliases
aliases = Codepagex.aliases()

# Get all possible aliases (including unconfigured)
all_aliases = Codepagex.aliases(:all)
```

--------------------------------

### Handle Invalid Sequences during Binary to String Conversion

Source: https://github.com/tallakt/codepagex/blob/main/README.md

Converts a binary to a string, using UTF-8 replacement characters for invalid sequences. This example demonstrates chaining operations.

```elixir
    iex> iso = "Hello æøå!" |> from_string!(:iso_8859_1)
    iex> to_string!(iso, :ascii, use_utf_replacement())
    "Hello !"
```

--------------------------------

### Convert UTF-8 to binary with custom handling using from_string!/4

Source: https://context7.com/tallakt/codepagex/llms.txt

Similar to from_string/4 but raises exceptions on errors.

```elixir
# Replace non-ASCII characters with underscore (raises on other errors)
result = Codepagex.from_string!("Hello æøå!", :ascii, Codepagex.replace_nonexistent("_"))
# Result: "Hello ___!"
```

--------------------------------

### List supported encodings

Source: https://context7.com/tallakt/codepagex/llms.txt

Retrieve a list of supported encodings, either configured or all available.

```elixir
# List currently configured encodings
encodings = Codepagex.encoding_list()

# List all available encodings (including unconfigured)
all_encodings = Codepagex.encoding_list(:all)
```

--------------------------------

### Configure encodings in config.exs

Source: https://context7.com/tallakt/codepagex/llms.txt

Define which encodings are available at runtime by updating the project configuration.

```elixir
# config/config.exs
import Config

# Include only ASCII encoding
config :codepagex, :encodings, [:ascii]

# Include multiple encodings using aliases, regex, or full names
config :codepagex, :encodings, [
  :ascii,                    # by alias name
  :iso_8859_1,              # by alias name
  ~r[iso8859]i,             # by regex matching full names
  "ETSI/GSM0338",           # by full name as string
  :"VENDORS/MICSFT/WINDOWS/CP1252"  # by full name as atom
]

# After changing configuration, recompile with:
# mix deps.compile codepagex --force
```

--------------------------------

### Convert String using Encoding Name as Atom

Source: https://github.com/tallakt/codepagex/blob/main/README.md

Demonstrates converting a string to a binary using an encoding specified as an atom. This is the preferred method for efficiency.

```elixir
    iex> from_string("æøå", :"ISO8859/8859-9")
    {:ok, <<230, 248, 229>>}
```

--------------------------------

### Convert String using Encoding Name as String

Source: https://github.com/tallakt/codepagex/blob/main/README.md

Demonstrates converting a string to a binary using an encoding specified as a string. The library handles the conversion to an atom internally.

```elixir
    iex> from_string("æøå", "ISO8859/8859-9")
    {:ok, <<230, 248, 229>>}
```

--------------------------------

### Convert String to Binary with ISO-8859-1 Encoding (Bang Version)

Source: https://github.com/tallakt/codepagex/blob/main/README.md

Converts a string to a binary representation using the ISO-8859-1 encoding, raising an error if conversion fails. This is a shorthand for the `from_string/2` function.

```elixir
    iex> from_string!("Hello æøåÆØÅ!", :iso_8859_1)
    <<72, 101, 108, 108, 111, 32, 230, 248, 229, 198, 216, 197, 33>>
```

--------------------------------

### Convert UTF-8 to binary with from_string!/2

Source: https://context7.com/tallakt/codepagex/llms.txt

Converts a UTF-8 string to a specified encoding, raising a Codepagex.Error on failure.

```elixir
# Convert UTF-8 string to ISO-8859-1 encoding (raises on error)
encoded = Codepagex.from_string!("æøåÆØÅ", :iso_8859_1)
# Result: <<230, 248, 229, 198, 216, 197>>

# This raises Codepagex.Error
Codepagex.from_string!("ʒ", :iso_8859_1)
# ** (Codepagex.Error) Invalid bytes for encoding
```

--------------------------------

### Convert binary to UTF-8 string with to_string!/2

Source: https://context7.com/tallakt/codepagex/llms.txt

Converts a binary in a specified encoding to a UTF-8 string, raising a Codepagex.Error on failure.

```elixir
# Convert ISO-8859-1 encoded binary to UTF-8 string (raises on error)
string = Codepagex.to_string!(<<72, 201, 166, 166, 211>>, :iso_8859_1)
# Result: "HÉ¦¦Ó"

# This raises Codepagex.Error
Codepagex.to_string!(<<128>>, "ETSI/GSM0338")
# ** (Codepagex.Error) Invalid bytes for encoding
```

--------------------------------

### Convert UTF-8 to binary with custom handling using from_string/4

Source: https://context7.com/tallakt/codepagex/llms.txt

Converts a UTF-8 string to a binary, using a custom function to handle unmappable characters.

```elixir
# Replace non-ASCII characters with underscore
{:ok, encoded, replacement_count} = Codepagex.from_string("Hello æøå!", :ascii, Codepagex.replace_nonexistent("_"))
# Result: {:ok, "Hello ___!", 3}

# With initial accumulator value for counting replacements
{:ok, encoded, total_count} = Codepagex.from_string("Hello æøå!", :ascii, Codepagex.replace_nonexistent("_"), 100)
# Result: {:ok, "Hello ___!", 103}

# Custom missing_fun implementation
missing_fun = fn encoding ->
  case Codepagex.from_string("#", encoding) do
    {:ok, replacement} ->
      inner_fun = fn <<_::utf8, rest::binary>>, acc ->
        {:ok, replacement, rest, acc + 1}
      end
      {:ok, inner_fun}
    err -> err
  end
end
{:ok, result, count} = Codepagex.from_string("Hello æøå!", :ascii, missing_fun, 0)
# Result: {:ok, "Hello ###!", 3}
```

--------------------------------

### to_string!/4

Source: https://context7.com/tallakt/codepagex/llms.txt

Converts a Codepagex binary to a string, raising an exception on errors. Supports custom missing_fun for handling invalid bytes.

```APIDOC
## to_string!/4

### Description
Converts a binary to a string, raising exceptions on errors. The missing_fun can prevent errors from occurring by handling all invalid bytes.

### Parameters
- **binary** (Codepagex.t) - Required - The binary to convert.
- **encoding** (atom/string) - Required - The target encoding.
- **missing_fun** (function) - Optional - A function to handle unmappable bytes.

### Request Example
Codepagex.to_string!(iso_binary, :ascii, Codepagex.use_utf_replacement())
```

--------------------------------

### Convert binary to UTF-8 string with to_string/2

Source: https://context7.com/tallakt/codepagex/llms.txt

Converts a binary in a specified encoding to a UTF-8 string, returning an error tuple on failure.

```elixir
# Convert ISO-8859-1 encoded binary to UTF-8 string
{:ok, string} = Codepagex.to_string(<<230, 248, 229, 198, 216, 197>>, :iso_8859_1)
# Result: {:ok, "æøåÆØÅ"}

# Using string encoding name
{:ok, string} = Codepagex.to_string(<<72, 201, 166, 166, 211>>, "ISO8859/8859-1")
# Result: {:ok, "HÉ¦¦Ó"}

# Invalid bytes return error
{:error, reason} = Codepagex.to_string(<<128>>, "ETSI/GSM0338")
# Result: {:error, "Invalid bytes for encoding"}
```

--------------------------------

### encoding_list/1

Source: https://context7.com/tallakt/codepagex/llms.txt

Retrieves a list of supported character encodings.

```APIDOC
## encoding_list/1

### Description
Returns a list of supported encodings. Call without arguments for configured encodings or with :all for all available encodings.

### Parameters
- **scope** (atom) - Optional - Use :all to list all available encodings.
```

--------------------------------

### Convert String to Binary with ISO-8859-1 Encoding

Source: https://github.com/tallakt/codepagex/blob/main/README.md

Converts a string to a binary representation using the specified encoding. Use the `from_string/3` variant to handle invalid byte sequences.

```elixir
    iex> from_string("æøåÆØÅ", :iso_8859_1)
    {:ok, <<230, 248, 229, 198, 216, 197>>}
```

```elixir
    iex> from_string!("æøåÆØÅ", :iso_8859_1)
    <<230, 248, 229, 198, 216, 197>>
```

--------------------------------

### Convert UTF-8 to binary with from_string/2

Source: https://context7.com/tallakt/codepagex/llms.txt

Converts a UTF-8 string to a specified encoding, returning an error tuple if characters are unmappable.

```elixir
# Convert UTF-8 string to ISO-8859-1 encoding
{:ok, encoded} = Codepagex.from_string("Hello æøåÆØÅ!", :iso_8859_1)
# Result: {:ok, <<72, 101, 108, 108, 111, 32, 230, 248, 229, 198, 216, 197, 33>>}

# Using full encoding name instead of alias
{:ok, encoded} = Codepagex.from_string("HÉ¦¦Ó", :"ISO8859/8859-1")
# Result: {:ok, <<72, 201, 166, 166, 211>>}

# Encoding characters not in target encoding returns error
{:error, reason} = Codepagex.from_string("ʒ", :iso_8859_1)
# Result: {:error, "Invalid bytes for encoding"}
```

--------------------------------

### Configure Codepagex Encodings in Mix Project

Source: https://github.com/tallakt/codepagex/blob/main/README.md

Specifies the encodings to be included in the codepagex compilation for a Mix project. This configuration is done in `config/config.exs`.

```elixir
    use Mix.Config
    config :codepagex, :encodings, [:ascii]
```

--------------------------------

### Convert Binary to String with ISO-8859-1 Encoding

Source: https://github.com/tallakt/codepagex/blob/main/README.md

Converts a binary representation back to a string using the specified encoding. Use the `to_string/3` variant to handle invalid byte sequences.

```elixir
    iex> to_string(<<230, 248, 229, 198, 216, 197>>, :iso_8859_1)
    {:ok, "æøåÆØÅ"}
```

```elixir
    iex> to_string!(<<230, 248, 229, 198, 216, 197>>, :iso_8859_1)
    "æøåÆØÅ"
```

--------------------------------

### Translate between encodings with exceptions

Source: https://context7.com/tallakt/codepagex/llms.txt

Perform encoding translation and raise a Codepagex.Error if the operation fails.

```elixir
# Translate between compatible encodings (raises on error)
result = Codepagex.translate!(<<174>>, :iso_8859_1, :iso_8859_15)

# This raises Codepagex.Error
Codepagex.translate!(<<174>>, :iso_8859_1, :iso_8859_2)
```

--------------------------------

### Convert binary to UTF-8 with custom handling using to_string/4

Source: https://context7.com/tallakt/codepagex/llms.txt

Converts a binary to UTF-8 with a custom function to handle invalid byte sequences.

```elixir
# Use UTF replacement character for invalid bytes
iso_binary = Codepagex.from_string!("Hello æøå!", :iso_8859_1)
{:ok, string, replacement_count} = Codepagex.to_string(iso_binary, :ascii, Codepagex.use_utf_replacement())
# Result: {:ok, "Hello !", 3}
```

--------------------------------

### Force Recompilation of Codepagex Dependency

Source: https://github.com/tallakt/codepagex/blob/main/README.md

Forces the recompilation of the codepagex dependency after modifying its encoding configuration in `config/config.exs`. This command is necessary because dependency configurations are not automatically detected.

```bash
mix deps.compile codepagex --force
```

--------------------------------

### Configure Multiple Codepagex Encodings with Regex and Strings

Source: https://github.com/tallakt/codepagex/blob/main/README.md

Configures codepagex to include multiple encodings using a list that contains aliases, regular expressions, and full encoding names as strings or atoms.

```elixir
    use Mix.Config
    config :codepagex, :encodings, [
      :ascii,
      ~r[iso8859]i,
      "ETSI/GSM0338",
      :"MISC/CP856"
    ]
```

--------------------------------

### translate/3

Source: https://context7.com/tallakt/codepagex/llms.txt

Converts a binary from one encoding to another, using UTF-8 as an intermediate step.

```APIDOC
## translate/3

### Description
Converts a binary from one encoding to another encoding. Internally converts to UTF-8 as an intermediate step. Returns {:ok, binary} on success.

### Parameters
- **binary** (binary) - Required - The input binary.
- **source_encoding** (atom/string) - Required - The source encoding.
- **target_encoding** (atom/string) - Required - The target encoding.

### Response
#### Success Response (200)
- **result** (binary) - The translated binary.
```

--------------------------------

### Convert to string with UTF replacement

Source: https://context7.com/tallakt/codepagex/llms.txt

Use to_string!/4 to convert binary data while automatically replacing invalid bytes with the UTF replacement character.

```elixir
# Use UTF replacement character (never raises with use_utf_replacement)
iso_binary = Codepagex.from_string!("Hello æøå!", :iso_8859_1)
result = Codepagex.to_string!(iso_binary, :ascii, Codepagex.use_utf_replacement())
```

--------------------------------

### Codepagex Encoding Functions

Source: https://context7.com/tallakt/codepagex/llms.txt

Functions for converting UTF-8 strings to binary encodings and vice versa, including variants that raise exceptions on failure.

```APIDOC
## Codepagex.from_string/2

### Description
Converts a UTF-8 Elixir string to a binary in the specified encoding.

### Parameters
- **string** (String) - The UTF-8 string to convert.
- **encoding** (Atom/String) - The target encoding (e.g., :iso_8859_1).

### Response
- **Success** ({:ok, binary}) - Returns the encoded binary.
- **Error** ({:error, reason}) - Returns an error tuple if characters cannot be represented.

## Codepagex.from_string!/2

### Description
Converts a UTF-8 Elixir string to a binary in the specified encoding, raising Codepagex.Error on failure.

## Codepagex.from_string/4

### Description
Converts a UTF-8 string to a binary with custom handling for unmappable characters using a missing_fun.

### Parameters
- **string** (String) - The UTF-8 string.
- **encoding** (Atom/String) - The target encoding.
- **missing_fun** (Function) - Logic for handling unmappable characters.
- **acc** (Any) - Initial accumulator value.

## Codepagex.to_string/2

### Description
Converts a binary in a specified encoding to a UTF-8 Elixir string.

### Response
- **Success** ({:ok, string}) - Returns the decoded UTF-8 string.
- **Error** ({:error, reason}) - Returns an error tuple if bytes cannot be decoded.

## Codepagex.to_string!/2

### Description
Converts a binary in a specified encoding to a UTF-8 string, raising Codepagex.Error on failure.

## Codepagex.to_string/4

### Description
Converts a binary in a specified encoding to UTF-8 with custom handling for invalid byte sequences.
```

--------------------------------

### Translate between encodings

Source: https://context7.com/tallakt/codepagex/llms.txt

Convert binary data between different encodings. Translation will fail if the character is not supported by the target encoding.

```elixir
# Translate from ISO-8859-1 to ISO-8859-15
{:ok, result} = Codepagex.translate(<<174>>, :iso_8859_1, :iso_8859_15)

# Translation fails if character not in target encoding
{:error, reason} = Codepagex.translate(<<174>>, :iso_8859_1, :iso_8859_2)
```

--------------------------------

### replace_nonexistent/1

Source: https://context7.com/tallakt/codepagex/llms.txt

Creates a missing_fun that replaces unmappable characters with a specified string.

```APIDOC
## replace_nonexistent/1

### Description
Returns a missing_fun for use with from_string/4 that replaces unmappable UTF-8 characters with the specified replacement string.

### Parameters
- **replacement** (string) - Required - The string to use for replacement.
```

--------------------------------

### Handle Invalid Sequences during String to Binary Conversion

Source: https://github.com/tallakt/codepagex/blob/main/README.md

Converts a string to a binary, replacing non-existent characters with a specified replacement character. It also returns the count of replacements made.

```elixir
    iex> from_string("Hello æøå!", :ascii, replace_nonexistent("_"))
    {:ok, "Hello ___!", 3}
```

--------------------------------

### Define custom missing_fun for to_string/4

Source: https://context7.com/tallakt/codepagex/llms.txt

Define a custom function to handle missing characters during encoding conversion. This function is passed to to_string/4 to replace invalid bytes.

```elixir
missing_fun = fn _encoding ->
  inner_fun = fn <<_, rest::binary>>, acc ->
    {:ok, "#", rest, acc + 1}
  end
  {:ok, inner_fun}
end
iso_binary = Codepagex.from_string!("Hello æøå!", :iso_8859_1)
{:ok, result, count} = Codepagex.to_string(iso_binary, :ascii, missing_fun, 10)
```

--------------------------------

### Replace nonexistent characters

Source: https://context7.com/tallakt/codepagex/llms.txt

Generate a missing_fun that replaces unmappable characters with a specific string.

```elixir
# Create a replacement function that uses underscore
missing_fun = Codepagex.replace_nonexistent("_")
{:ok, result, count} = Codepagex.from_string("Café résumé", :ascii, missing_fun)

# Use question mark as replacement
missing_fun = Codepagex.replace_nonexistent("?")
result = Codepagex.from_string!("naïve", :ascii, missing_fun)
```

--------------------------------

### Use Unicode replacement character

Source: https://context7.com/tallakt/codepagex/llms.txt

Generate a missing_fun that replaces invalid bytes with the Unicode replacement character (U+FFFD).

```elixir
# Decode binary with invalid bytes replaced by replacement character
iso_binary = Codepagex.from_string!("Héllo", :iso_8859_1)
{:ok, result, count} = Codepagex.to_string(iso_binary, :ascii, Codepagex.use_utf_replacement())

# Chain operations with error handling
iso_binary = <<72, 233, 108, 108, 111>>  # "Héllo" in ISO-8859-1
result = Codepagex.to_string!(iso_binary, :ascii, Codepagex.use_utf_replacement())
```
Tokens:
Raw