This repository has been archived by the owner on Jun 1, 2023. It is now read-only.
Component fields in 5.x dictionaries not accommodated #4
Assignees
Labels
Comments
|
Real crux of issue appears to be varying group definitions (of the same group type, e.g. NoLegs) across different messages. So NoLegs as part of InstrumentLegGrp may have a different # of supported fields compared to NoLegs in QuotReqLegsGrp. fix2json is not grouping groups by message type, so the internal data dictionary representation entry for any particular group name key always reflects the last instance of NoLegs encountered in the dictionary file. There is some evidence in support of this being a classic rite of passage on the FIX parsing circuit. |
|
I am having a similar issue it seems like the when the message entry type changes from "TRADE" to "BID/OFFER" before the last entry on the array the last sub-message is left out of the array. Other Message |
|
Yes, indeed this looks to be stemming from the same fundamental issue. There have been a few steps taken toward fixing this but unfortunately there's still a little bit more work to be done. Stay tuned and we'll update the status on this issue as soon as a branch is created with a candidate fix. |
|
I checked some code into the grouptreatment branch that appears to fix this issue for MarketDataIncrementalRefresh but still seems to mis-process SecurityDefinitions. You might want to give it a shot for your purposes if limited to MarketDataIncrementalRefresh's immediately. Turns out that much of the issue stemmed from additional required fields in the data dictionary for CME. Not sure if there is a canonical dictionary in QuickFix format that CME provides to members as part of its SDK, but if there is it might forestall future complications of this kind. I recall seeing some extension packs somewhere on FIX Protocol's official site but have not had a chance to delve into those quite yet. After checking out the branch, you should see output resembling: Please let us know how the new code works out for you. Thanks! Sal |
|
Thank you Sal, I will try to follow up with the CME data people about the canonical dictionary for QuickFix format. Now I cloned and checkout the git repository and I am getting the following error. Any suggestions on how to fix it? |
|
Hi jlroo, Were you using the If you could share the full command that you were executing when you encountered that error, that'd be helpful as well. Thanks! Sal |
|
Sal, I did run install and switched branches. Below are all the commands that I used. Thank you! |
|
Thanks for sending that over. I executed the identical commands and received different results: Not entirely sure what might be contributing to this. I tested this with: Would you be able to check your node version and/or perhaps try executing on a different machine to see if this might be related to an environment discrepancy issue? Thanks! Sal |
|
Sal thank you for all the support. Its definitely a problem with the node version. It works fine with node v4.2.6 on CentOS 6.5 Yum Nodejs - CentOS 6.5 Nodejs from tar - CentOS 6.5 |
|
Hi @jlroo, I'm actually not so sure now whether it is the node version or not. Last night I ran a whole Datamine sample file through and before it was complete, it failed with the following error: That's just too much of a coincidence. Will isolate the failing record, but I suspect there's a field in the patched data dictionary still missing that the group parser is failing on. Does it stop for you right away or does it successfully process a number of records before the exception? |
|
Sal, I experienced the same issue in two different machines and versions of node. Initially I thought that the problem was just on CentOS 6.5 but I get the same error on my mac. Here are the three fix messages right after "MsgSeqNum": 1550140 the last processed message before getting the error. |
salsferrazza
pushed a commit
that referenced
this issue
Feb 9, 2016
|
Hi @jlroo, You can try grabbing the grouptreatment branch again. I took those FIX messages above and put them into testfiles/issue#4.txt, they now seem to be parsing out correctly. Are these the results you are expecting from that input? |
|
Sal those messages are correct. I searched for other messages with errors in the data previous the group treatment and I found a duplicated message (same "MsgSeqNum") one with the correct groups, the other with the problem. I did the same for the new version and this issues seems to be corrected Here are the messages: Old Version New Version |
|
OK great - I will likely merge this into the master branch sometime soon, but am still looking into how fix2json treats repeating groups within message types other than MarketData* (e.g. SecurityDefinition). |
|
Sal, Last Two json Msgs Last processed FIX Msg Next Three Msg |
|
Hi Jlroo, Thanks for sending this over, will pull down those new datasets and let you know if I can see anything. Sal |
|
Hi jlroo, I've updated the CME data dictionary and the error is gone although the parser seems to be omitting the value of the final OrderID in the last member of the repeating group NoOrderIDEntries. Can you have a look and let me know if it's better for the data sets you are running fix2json against? The parser is also putting NoOrderIDEntries underneath the NoMDEntries group, but it should be moved up one level, obviously based on the Merc's own MDP FIX documentation. Still working on this but may track it with a separate issue. Stay tuned. But in the interim you can just update the FIX50SP2.CME.xml dictionary currently committed into the master branch. |
|
Sal, Thank you for all your great work. In the future if I encounter this error I will refer to the dictionary. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
When using 5.x FIX dictionaries (e.g. processing CME Datamine historicals), component field definitions under repeating groups are ignored and thus groups containing sub-components (e.g. NoLegs) in their dictionary schemae are truncated and yield a semantically incorrect interpretation of the inbound FIX message.
The dictionary parser does not go within the 'InstrumentLeg' component below to catalog its fields for inclusion within group field candidacy (necessary for the algorithm to determine when it has reached the end of a repeating group in the source FIX).
The dictionary code could use a bit of refactoring out of the main source file and into its own module. It might make sense to approach the resolution to this issue with that in mind. Solution likely requires a recursive function in processing the DOM nodes since components can have components as components.
The text was updated successfully, but these errors were encountered: