Parsing and Validating an IDoc Flat File

Goal

You break an SAP IDoc flat file into control and data records, parse and validate SDATA according to the segment layout, and convert it into an internal standard JSON.

Why it matters

In large Korean enterprise projects, you almost always meet ERP integration. And the moment "you can receive it with IDoc" comes up in a meeting, if you do not know the structure you cannot decide anything. In reality, IDoc parsing is finding the layout by the segment name and cutting the 1000-character SDATA by position, the same work as the fixed-length parsing in the previous module. And just knowing that status code 51, which you will see most often in operation, is not "transmission failure" but "business processing failure inside SAP" can break a deadlock between integration owners once.

Steps

1. Source: /opt/lab/fixtures/eai/idoc/ORDERS05.txt, layout: /opt/lab/fixtures/eai/idoc/segments.md
2. Create /root/idoc and split out only the control records to /root/idoc/control.txt and only the data records to /root/idoc/data.txt. The sum of the line counts of the two files must equal the line count of the source.
3. Create /root/idoc/ctrl.csv. The first line is docnum,idoctyp,mestyp,sndprn,rcvprn,credat. One row per control record, in ascending docnum order.
4. Create /root/idoc/segstat.csv. The first line is segnam,count. List the count per segment name in descending count order, ties by ascending name.
5. Parse the E1EDK01 segment and create /root/idoc/header.csv. The first line is docnum,belnr,curcy,netwr. netwr is the header total.
6. Parse the E1EDP01 segment and create /root/idoc/items.csv. The first line is docnum,posex,matnr,menge,netpr,amount. amount is menge × netpr.
7. Create /root/idoc/validate.sh. It runs with no arguments, and for each IDoc in which the header's netwr and the total of the item amount differ, it prints the docnum, one per line, and ends with a non-zero exit code if there is even one. Save the result of running it to /root/idoc/mismatch.txt. (There is 1 case.)
8. Create /root/idoc/status.csv. The first line is code,meaning,action. Include all six codes 03, 12, 51, 53, 64, and 68, and action is one of 정상 (normal), 대기 (wait), 조사 (investigate), or 재전송 (resend).
9. Create /root/idoc/orders.json. The top level is an array, and each element has the structure below. In ascending docnum order.

   { "docnum": "...", "belnr": "...", "currency": "...",
     "netAmount": <숫자>,
     "items": [ { "posex": "...", "matnr": "...",
                  "qty": <숫자>, "price": <숫자> }, ... ] }
   

Notes

• Fixed-length cutting is convenient with Python slicing: line[10:30] (the positions in the layout are 1-based, so subtract 1 for the index)
• Checking the structure with jq: jq '.[0].items | length' orders.json
• Common mistake 1: using the layout's 1-based positions as 0-based indexes as is.
• Common mistake 2: not deciding how to handle leading zeros and decimal points in quantity and amount fields.
• Common mistake 3: interpreting status code 51 as "transmission failure." The data arrived, and business processing failed inside SAP.

Split control and data records

Create /root/idoc and split out only the control records to /root/idoc/control.txt and only the data records to /root/idoc/data.txt. The sum of the line counts of the two files must equal the line count of the source.

The record type is marked as a separator at the beginning of each line. Check the file structure by eye first and then decide the splitting criterion.

Parse the control record

Create /root/idoc/ctrl.csv. The first line is docnum,idoctyp,mestyp,sndprn,rcvprn,credat. One row per control record, in ascending docnum order.

The control record holds what this document is and from whom to whom it goes. You must cut by the positions in the layout specification.

Segment statistics

Create /root/idoc/segstat.csv. The first line is segnam,count. List the count per segment name in descending count order, ties by ascending name.

If you count how many of each segment name there are, the document structure shows. Header-type segments should be 1 and item-type segments should be several.

Parse the header segment

Parse the E1EDK01 segment and create /root/idoc/header.csv. The first line is docnum,belnr,curcy,netwr. netwr is the header total.

SDATA is a fixed-length string. Each segment has a different layout, so you must find that segment in the specification.

Parse the item segment

Parse the E1EDP01 segment and create /root/idoc/items.csv. The first line is docnum,posex,matnr,menge,netpr,amount. amount is menge × netpr.

There are several items. If you extract each one's quantity and unit price and compute the amount in advance, you will use it in the next step.

Header-to-item amount validation

Create /root/idoc/validate.sh. It runs with no arguments, and for each IDoc in which the header's netwr and the total of the item amount differ, it prints the docnum, one per line, and ends with a non-zero exit code if there is even one. Save the result of running it to /root/idoc/mismatch.txt. (There is 1 case.)

If the total written in the header differs from the sum of the items, that document must not be processed. Tell us which IDoc is the problem by its number.

Status code mapping table

Create /root/idoc/status.csv. The first line is code,meaning,action. Include all six codes 03, 12, 51, 53, 64, and 68, and action is one of 정상 (normal), 대기 (wait), 조사 (investigate), or 재전송 (resend).

"Transmission failure" and "business processing failure inside SAP" are different. If you cannot tell them apart, the owners on both sides run in parallel lines.

Convert to the internal standard JSON

Create /root/idoc/orders.json. The top level is an array, and each element has the structure below. In ascending docnum order.

{ "docnum": "...", "belnr": "...", "currency": "...",
  "netAmount": <숫자>,
  "items": [ { "posex": "...", "matnr": "...",
               "qty": <숫자>, "price": <숫자> }, ... ] }

Preserve the segments' hierarchy level and put the items in an array under the header. Make it so the structure can be verified with jq.