A Translation Adapter Between Fixed-Length Messages and JSON

Goal

Build converters that read a layout file and a code mapping table and convert between fixed-length bodies (EUC-KR) and JSON, and make them reject values that violate the rules for byte length, code, encoding and decimals.

Why it matters

Mistakes in the transformation layer are quiet. A length counted in characters shifts all the following fields, passing along an unknown code lets the internal system read it with a different meaning, a rate that went through float is short by 1, and Python's euc_kr converts the syllable U+B620 into 8 bytes and passes it on. So you keep the rules as data (layout, mapping table), and for values that cannot be fixed, you do not fix them but reject them with an error code.

Steps

1. Transcribe the specification /opt/lab/fixtures/eaimw/transform/FXTR0001.md into /root/eaimw/xform/FXTR0001.layout. Header name,length,type,key,map,scale, 12 rows in the order of the specification. name is the English name, key is the JSON key, map is the code mapping domain (blank if none), and scale is blank in this message.
2. Transcribe /opt/lab/fixtures/eaimw/transform/CODES.md into /root/eaimw/xform/codemap.csv. Header domain,external,internal, only codes in use, sorted by domain then external (LC_ALL=C sort).
3. Create /root/eaimw/xform/f2j.py. python3 f2j.py <레이아웃> <매핑표> <본문파일> (layout, mapping table, body file) prints one line of JSON and exits with 0. N is an integer, AN and H have only trailing spaces removed, the body is read strictly as EUC-KR, and fields with a map are converted with the mapping table. When rejecting, start the first line of standard error with the error code and use exit code 2: a code not in the mapping table UNMAPPED; body length mismatch, non-numeric, or unreadable as EUC-KR BAD_VALUE.
4. Create /root/eaimw/xform/j2f.py. python3 j2f.py <레이아웃> <매핑표> <JSON파일> (layout, mapping table, JSON file) outputs the body bytes to standard output. N is right-aligned with leading zeros, AN and H are left-aligned with trailing spaces, and length is in EUC-KR bytes. Rejections (exit code 2): length or digit overflow OVERFLOW (do not truncate), an internal code not in the mapping table UNMAPPED, a missing key or N that is not an integer of 0 or more BAD_VALUE.
5. Create /root/eaimw/xform/roundtrip.py. python3 roundtrip.py <레이아웃> <매핑표> <본문파일>... (layout, mapping table, body files) prints one line <파일이름>,<결과> (file name, result) per file — REJECT if f2j rejects, SAME if the f2j→j2f result equals the original bytes, DIFF if it differs. Then save the results for all of the inbox's FXTR0001-*.body (in name order), with the header file,result, to /root/eaimw/xform/roundtrip.csv.
6. Transcribe the specification /opt/lab/fixtures/eaimw/transform/DPRT0001.md into /root/eaimw/xform/DPRT0001.layout (scale 4 for implicit-decimal fields), and fix f2j and j2f to support scale. f2j gives a string written out to the scale places, like "3.2500", and j2f accepts only strings and calculates with decimal. If there are more decimal places than the scale, OVERFLOW (no rounding); if it is not a string (including a JSON number), BAD_VALUE.
7. Fix j2f to reject characters outside the KS X 1001 precomposed set (for example the syllable U+B620, emoji) with NOT_KSX1001 (exit code 2). Characters that are 2 bytes in EUC-KR, such as Chinese characters and symbols, are accepted as is.
8. Convert the whole inbox /opt/lab/fixtures/eaimw/transform/inbox/ with f2j. The first part of the file name (FXTR0001, DPRT0001) is the layout name. On success, /root/eaimw/xform/out/<이름>.json (the name with the .body extension removed); if rejected, leave no JSON. Write the results with the header file,result, in file name order, the result being OK or an error code, to /root/eaimw/xform/summary.csv.

Notes

• Working with bytes: read the body with open(f, "rb").read(), fields with body[pos:pos + 길이] (length), and when converting to characters .decode("euc_kr"). For the length check, len(값.encode("euc_kr")) (value).
• Error output: print("UNMAPPED ...", file=sys.stderr); sys.exit(2). Byte output: sys.stdout.buffer.write(본문) (body).
• j2f can borrow f2j's load_layout and load_codemap (from f2j import ... — files in the same directory are imported as they are).
• Common mistakes: counting length in characters, truncating an overflowing value and passing it on, passing an unknown code through as it is, multiplying an interest rate as a float.
• The grader creates a new layout, mapping table and values each time and passes them as arguments. If you write the specification's field names in the code, you cannot pass.
• The common library /opt/lab/fixtures/eaimw/lib/lhconv.py is used from module 4. In this lab you build it yourself.

Transcribe the specification into a layout file

Transfer the 12 fields of /opt/lab/fixtures/eaimw/transform/FXTR0001.md into /root/eaimw/xform/FXTR0001.layout (name,length,type,key,map,scale).

Transcribe the table in section 1 of the specification as it is. name is the English name, key is the JSON key (case as written), and map is the code mapping column. Adding up all the lengths must equal the business-part total stated in the specification.

Transcribe the code definitions into a mapping table

Transfer the codes in use from /opt/lab/fixtures/eaimw/transform/CODES.md into /root/eaimw/xform/codemap.csv (domain,external,internal) and sort by domain then external.

The domains are the three BANK, CHANNEL and KIND. Do not load rows whose status is discontinued — that way a message containing such a code gets caught in conversion. The leading 0 of external code 01 is part of the value (the 0 that disappears when you open it in Excel).

Fixed-length → JSON

/root/eaimw/xform/f2j.py prints JSON, and rejects unknown codes with UNMAPPED and format or encoding errors with BAD_VALUE (exit code 2).

Read with open(file, 'rb') and cut the bytes at the layout's lengths. If you decode('euc_kr') each field, half-Hangul and CP949-only characters show up as UnicodeDecodeError. For N, check that it is numeric after strip and convert to int; for AN and H, do only rstrip(' '). Do not pass a code not in the mapping table through as it is.

JSON → fixed-length, counted in bytes

/root/eaimw/xform/j2f.py outputs the body bytes, and rejects a length overflow with OVERFLOW, an unknown internal code with UNMAPPED, and a missing key or non-integer with BAD_VALUE.

The length of an H field is not len(value) but len(value.encode('euc_kr')). If it overflows, do not cut it to fit; reject it — cutting by bytes splits Hangul in half. Look up the mapping in a dictionary that inverts the mapping table (internal → external). For N use rjust(length, '0'), and for the rest ljust(length, b' ').

Round-trip test

/root/eaimw/xform/roundtrip.py sorts each body into SAME, DIFF or REJECT, and saves the results for all the FXTR0001 samples in the inbox to /root/eaimw/xform/roundtrip.csv.

Import f2j and j2f as functions and use them. If f2j rejects, it is REJECT, and if the restored bytes equal the original, it is SAME. A sample that comes out DIFF may be a converter bug, but it may also be the peer violating the standard — look at the bytes of that field with od -c.

Implicit decimals with decimal

Create /root/eaimw/xform/DPRT0001.layout (scale 4), and fix f2j to give a string like "3.2500" and j2f to accept only string decimals and calculate with decimal.

Reading 0032500 with scale 4 gives Decimal(32500).scaleb(-4) = 3.2500. To convert back, Decimal("3.25").scaleb(4) = 32500. float("0.29") * 100 is 28.999999999999996, so converting it to int gives 28. If there are more decimal places than the scale, do not round; reject it.

Reject characters outside the precomposed set

Fix j2f to reject characters not in KS X 1001 precomposed Hangul (the syllable U+B620, emoji) with NOT_KSX1001 (exit code 2), and to accept 2-byte characters such as Chinese characters and symbols.

First try encoding the syllable U+B620 with python3 -c "print(chr(0xB620).encode('euc_kr'))" — you get not an error but 8 bytes. It is enough to check for each character whether the length of encode('euc_kr') is 2. Characters for which encoding itself fails (emoji) are also rejected with the same code.

Convert the whole inbox

Convert every body in the inbox with f2j, leaving the successful ones as /root/eaimw/xform/out/.json and the list of results in /root/eaimw/xform/summary.csv (file,result).

The part of the file name before '-' is the layout name. With exit code 2, the first word of the first line of standard error is the error code. Shell redirection leaves an empty file even on failure, so delete the JSON files of rejected samples.