TT Lab
Get started
Learn Learning paths Courses

Insurance Domain Deep Dive

Finding the Receipt That Was Claimed Twice

Continue in TT Lab

Goal

You build a detector, dedup.py, that separates exact duplicates from de facto duplicates among received claims and produces judgments, with their basis, that exclude the false positive of different treatments at the same hospital on the same day.

Why it matters

It is an everyday occurrence, not an exception, for the same single medical visit to come in several times through the app, web, and counter. A resubmission with the same claim number ends once grouped, but a de facto duplicate with a different claim number and only wobbling notation shows up only after normalization. If you miss it, the same money goes out twice. An accident in the opposite direction is quieter. If you block as a duplicate the claim of a person who really received treatment twice at the same hospital on the same day, the customer cannot get paid without knowing why. Half the work of building a detector is reducing this false positive, and so a basis is always attached to the judgment. The grader does not trust your judgments. It sets up a temporary claim table, runs your dedup.py, and checks the normalization result, blocks, scores, and judgments against the values the grader computes. The patients, hospitals, receipt numbers, and amounts change with each run.

Steps

  1. Create and run /root/dupclaim/gen_claims.py to make a day's worth of intake in /root/dupclaim/claims.db.
  2. Pull the resubmissions with the same claim number into /root/dupclaim/exact_dup.csv.
  3. Have /root/dupclaim/dedup.py normalize the patient, hospital, and receipt and compute the block key.
  4. Have dedup.py build blocks to reduce the pairs to compare.
  5. Have dedup.py assign similarity, a combined score, and a score grade to every pair within a block.
  6. Put the amount and date tolerances into dedup.py.
  7. Have dedup.py take consecutive-number receipts out as false positives and produce the final verdict and basis.
  8. Judge with your own claim table and leave /root/dupclaim/dedup_result.json, /root/dupclaim/review.csv, and /root/dupclaim/dup_report.md.

Notes

Generate a day's worth of intake

Create and run /root/dupclaim/gen_claims.py to build /root/dupclaim/claims.db. The claim table has 261 rows and 250 distinct claim numbers. Plant 10 resubmissions with the same claim number (one of them came in three times), 12 pairs whose patient, hospital, and receipt become exactly identical after normalization but whose original notation differs, and 6 pairs whose receipts are consecutive numbers within the same block, and use three values for channel: app, web, counter.

Four kinds of notation wobble are plenty: inserting spaces, turning the hyphen full-width, writing digits full-width, and writing in lowercase. For a consecutive pair, just raise the last digit of the receipt number by 1 for the same patient, hospital, and service date. You have to keep the variety of hospitals and service dates narrow for two or more to gather in a block.

Group the resubmissions with the same claim number first

In /root/dupclaim/exact_dup.csv, pull the cases where the same claim number came in two or more times, with the header claim_no,copies,first_received,last_received. One line per claim number.

Read /root/dupclaim/claims.db made in step 1. GROUP BY with HAVING COUNT(*) > 1 is all it takes. first_received and last_received are the minimum and maximum of the receipt times that came in under that claim number, and the number of resubmissions is not always 2. These are resubmissions, not de facto duplicates, so they are not counted as duplicate cases.

Fold notation wobble away by normalization

Have /root/dupclaim/dedup.py fold resubmissions into one line per claim number, compute each claim's patient_n, hospital_n, receipt_n, and block, and put them into normalized of the result JSON.

unicodedata.normalize("NFKC", s) folds full-width alphanumerics and the full-width hyphen into ordinary characters. Then remove whitespace, and for receipts keep only alphanumerics and uppercase them. When folding, keep the line with the earliest receipt time.

Reduce the pairs to compare with blocks

Have /root/dupclaim/dedup.py produce blocks. The key is the block key and the value is an array of the claim numbers belonging to that block sorted in ascending order, and put in only the blocks with two or more members.

If you use the pre-normalization hospital name in the block key, pairs with wobbling notation split into different blocks and never become candidates to begin with. Comparing all 250,000 gives 30 billion pairs, so blocking is a matter not of performance but of feasibility.

Assign similarity and a combined score

Have /root/dupclaim/dedup.py assign name_sim, receipt_sim, hospital_same, score, and score_verdict to every pair within a block and put them into pairs.

difflib.SequenceMatcher(None, x, y).ratio() is 2.0*M/T for the total element count T and the matching element count M of the two sequences. Round at the fourth decimal place. In pairs, in ascending claim number order a comes before b.

Tolerances for amount and date

Have /root/dupclaim/dedup.py attach amount_gap, day_gap, and in_tolerance to each pair. in_tolerance is true only when the amount difference is 100 or less and the service date difference is 0.

Even if the score is 1.0, if the amount is thousands of won apart it may not be the same receipt. Read the service date with datetime.date.fromisoformat and subtract to get the number of days. You look at the service date, not the filing date.

Take out the false positive of consecutive-number receipts

Have /root/dupclaim/dedup.py attach serial_neighbor, the final verdict, and reasons to each pair. A consecutive-number receipt is distinct even if the score is high.

If only one character of a twelve-digit number differs, the similarity exceeds 0.93. Since the name and hospital are also the same, the combined score passes the threshold, but this is not the same document but an adjacent one. Split off only the last chunk of digits and compare.

Produce output the review staff can look at as it is

Judge with your own claim table and leave /root/dupclaim/dedup_result.json, and write the pairs whose verdict is not distinct to /root/dupclaim/review.csv with the header a,b,verdict,score,amount_gap,day_gap,reasons (join reasons with a vertical bar). In /root/dupclaim/dup_report.md write five sections, ## 판정 요약 ## 완전 중복과 사실상 중복 ## 사람이 봐야 하는 건 ## 오탐으로 뺀 것 ## 정규화 규칙 (the five Korean section titles mean: judgment summary, exact and de facto duplicates, cases a person must look at, what was taken out as false positives, and normalization rules), and in the summary write the pair counts of duplicate, review, distinct, and serial_neighbor as a table.

Just feed /root/dupclaim/claims.db to the /root/dupclaim/dedup.py you built in the earlier steps. The review list is a file that the person in charge opens in a spreadsheet. With only a score they do not know what to check, so put the basis in too. In the false-positive section you must write the claim numbers of the pairs actually taken out, so that the next person can verify the rule.