Make bulk import random walk verify step print missing markers

[keith-turner]

Make bulk import random walk verify step print missing markers

[Manno15]

My current solution is to create a string builder to hold the thrown error message and then actually throw the error later after the marker error would normally get thrown. That way, the user would have the most complete information set. The initial thrown error for a bad key would cause the errors below to trigger since it is dependent on the marker number (which one would be missing). Thoughts on this approach @keith-turner @DomGarguilo? Another approach would do the row iterator first but I assume similar issues would arise.

[keith-turner]

My current solution is to create a string builder to hold the thrown error message and then actually throw the error later after the marker error would normally get thrown.

That sounds good. Another approach to consider is instead of using the string builder, could log information and then throw an exception with a shorter message after logging. I would go with whatever approach seems easiest to read, but not sure which one is.

[keith-turner]

This is what the row looks like in accumulo

root@uno> table bulk_dev_uk_1088323_1665128094663
root@uno bulk_dev_uk_1088323_1665128094663> scan -r r00000 -np
r00000 cf:000 []	-1
r00000 cf:001 []	-1
r00000 cf:002 []	-1
r00000 cf:003 []	-1
r00000 cf:004 []	-1
r00000 cf:005 []	-1
r00000 cf:006 []	-1
r00000 cf:007 []	-1
r00000 cf:008 []	-1
r00000 cf:009 []	-1
r00000 marker:0000001 []	1
r00000 marker:0000002 []	1
r00000 marker:0000003 []	1
r00000 marker:0000004 []	1
r00000 marker:0000005 []	1
r00000 marker:0000006 []	1
r00000 marker:0000007 []	1
r00000 marker:0000008 []	1
r00000 marker:0000009 []	1
r00000 marker:0000010 []	1
r00000 marker:0000011 []	1
r00000 marker:0000012 []	1
r00000 marker:0000013 []	1
r00000 marker:0000014 []	1
r00000 marker:0000015 []	1
r00000 marker:0000016 []	1
r00000 marker:0000017 []	1
r00000 marker:0000018 []	1
r00000 marker:0000019 []	1
r00000 marker:0000020 []	1
r00000 marker:0000021 []	1
r00000 marker:0000022 []	1
r00000 marker:0000023 []	1
r00000 marker:0000024 []	1
r00000 marker:0000025 []	1
r00000 marker:0000026 []	1
r00000 marker:0000027 []	1
r00000 marker:0000028 []	1
r00000 marker:0000029 []	1
r00000 marker:0000030 []	1
r00000 marker:0000031 []	1
r00000 marker:0000032 []	1
r00000 marker:0000033 []	1
r00000 marker:0000034 []	1
r00000 marker:0000035 []	1
r00000 marker:0000036 []	1
r00000 marker:0000037 []	1
r00000 marker:0000038 []	1
r00000 marker:0000039 []	1
r00000 marker:0000040 []	1
r00000 marker:0000041 []	1
r00000 marker:0000042 []	1
r00000 marker:0000043 []	1
r00000 marker:0000044 []	1
r00000 marker:0000045 []	1
r00000 marker:0000046 []	1
r00000 marker:0000047 []	1
r00000 marker:0000048 []	1
r00000 marker:0000049 []	1
r00000 marker:0000050 []	1
r00000 marker:0000051 []	1
r00000 marker:0000052 []	1
r00000 marker:0000053 []	1
r00000 marker:0000054 []	1
r00000 marker:0000055 []	1
r00000 marker:0000056 []	1
r00000 marker:0000057 []	1
r00000 marker:0000058 []	1
r00000 marker:0000059 []	1
r00000 marker:0000060 []	1
r00000 marker:0000061 []	1
r00000 marker:0000062 []	1
r00000 marker:0000063 []	1
r00000 marker:0000064 []	1
r00000 marker:0000065 []	1
r00000 marker:0000066 []	1
r00000 marker:0000067 []	1
r00000 marker:0000068 []	1
r00000 marker:0000069 []	1
r00000 marker:0000070 []	1
r00000 marker:0000071 []	1
r00000 marker:0000072 []	1
r00000 marker:0000073 []	1
r00000 marker:0000074 []	1
r00000 marker:0000075 []	1
r00000 marker:0000076 []	1
r00000 marker:0000077 []	1
r00000 marker:0000078 []	1
r00000 marker:0000079 []	1
r00000 marker:0000080 []	1
r00000 marker:0000081 []	1
r00000 marker:0000082 []	1
r00000 marker:0000083 []	1
r00000 marker:0000084 []	1
r00000 marker:0000085 []	1
r00000 marker:0000086 []	1
r00000 marker:0000087 []	1
r00000 marker:0000088 []	1
r00000 marker:0000089 []	1
r00000 marker:0000090 []	1
r00000 marker:0000091 []	1
r00000 marker:0000092 []	1
r00000 marker:0000093 []	1
r00000 marker:0000094 []	1
r00000 marker:0000095 []	1
r00000 marker:0000096 []	1
r00000 marker:0000097 []	1
r00000 marker:0000098 []	1
r00000 marker:0000099 []	1
r00000 marker:0000100 []	1
r00000 marker:0000101 []	1
r00000 marker:0000102 []	1
r00000 marker:0000103 []	1
r00000 marker:0000104 []	1
r00000 marker:0000105 []	1
r00000 marker:0000106 []	1
r00000 marker:0000107 []	1
r00000 marker:0000108 []	1
r00000 marker:0000109 []	1
r00000 marker:0000110 []	1
r00000 marker:0000111 []	1
r00000 marker:0000112 []	1
r00000 marker:0000113 []	1
r00000 marker:0000114 []	1
r00000 marker:0000115 []	1
r00000 marker:0000116 []	1
r00000 marker:0000117 []	1
r00000 marker:0000118 []	1
r00000 marker:0000119 []	1
r00000 marker:0000120 []	1
r00000 marker:0000121 []	1
r00000 marker:0000122 []	1
r00000 marker:0000123 []	1
r00000 marker:0000124 []	1
r00000 marker:0000125 []	1
r00000 marker:0000126 []	1
r00000 marker:0000127 []	1
r00000 marker:0000128 []	1
r00000 marker:0000129 []	1
r00000 marker:0000130 []	1
r00000 marker:0000131 []	1
r00000 marker:0000132 []	1
r00000 marker:0000133 []	1
r00000 marker:0000134 []	1
r00000 marker:0000135 []	1
r00000 marker:0000136 []	1
r00000 marker:0000137 []	1
r00000 marker:0000138 []	1
r00000 marker:0000139 []	1
r00000 marker:0000140 []	1
r00000 marker:0000141 []	1
r00000 marker:0000142 []	1
r00000 marker:0000143 []	1
r00000 marker:0000144 []	1
r00000 marker:0000145 []	1
r00000 marker:0000146 []	1
r00000 marker:0000147 []	1
r00000 marker:0000148 []	1
r00000 marker:0000149 []	1
r00000 marker:0000150 []	1
r00000 marker:0000151 []	1
r00000 marker:0000152 []	1
r00000 marker:0000154 []	1
r00000 marker:0000155 []	1
r00000 marker:0000156 []	1
r00000 marker:0000157 []	1
r00000 marker:0000158 []	1
r00000 marker:0000159 []	1
r00000 marker:0000160 []	1
r00000 marker:0000161 []	1
r00000 marker:0000162 []	1
r00000 marker:0000163 []	1
r00000 marker:0000164 []	1
r00000 marker:0000165 []	1
r00000 marker:0000166 []	1
r00000 marker:0000167 []	1
r00000 marker:0000168 []	1
r00000 marker:0000169 []	1
r00000 marker:0000170 []	1
r00000 marker:0000171 []	1
r00000 marker:0000172 []	1
r00000 marker:0000173 []	1
r00000 marker:0000174 []	1
r00000 marker:0000175 []	1
r00000 marker:0000176 []	1
r00000 marker:0000177 []	1
r00000 marker:0000178 []	1
r00000 marker:0000179 []	1
r00000 marker:0000180 []	1
r00000 marker:0000181 []	1
r00000 marker:0000182 []	1
r00000 marker:0000183 []	1
r00000 marker:0000184 []	1
r00000 marker:0000185 []	1
r00000 marker:0000186 []	1
r00000 marker:0000187 []	1
r00000 marker:0000188 []	1
r00000 marker:0000189 []	1
r00000 marker:0000190 []	1
r00000 marker:0000191 []	1
r00000 marker:0000192 []	1
r00000 marker:0000193 []	1
r00000 marker:0000194 []	1
r00000 marker:0000195 []	1
r00000 marker:0000196 []	1
r00000 marker:0000197 []	1
r00000 marker:0000198 []	1
r00000 marker:0000199 []	1
r00000 marker:0000200 []	1
r00000 marker:0000201 []	1
r00000 marker:0000202 []	1
r00000 marker:0000203 []	1
r00000 marker:0000204 []	1
r00000 marker:0000205 []	1
r00000 marker:0000206 []	1
r00000 marker:0000207 []	1
r00000 marker:0000208 []	1
r00000 marker:0000209 []	1
r00000 marker:0000210 []	1
r00000 marker:0000211 []	1
r00000 marker:0000212 []	1
r00000 marker:0000213 []	1
r00000 marker:0000214 []	1
r00000 marker:0000215 []	1
r00000 marker:0000216 []	1
r00000 marker:0000217 []	1
r00000 marker:0000218 []	1
r00000 marker:0000219 []	1
r00000 marker:0000220 []	1
r00000 marker:0000221 []	1
r00000 marker:0000222 []	1

[Manno15]

@keith-turner I can take a look into this if you aren't already.

[Manno15]

@keith-turner Did you alter any settings in order to get Bulk.xml to fail more consistently?

[keith-turner]

@keith-turner I can take a look into this if you aren't already.

Yeah go ahead I am not working on it.

@keith-turner Did you alter any settings in order to get Bulk.xml to fail more consistently?

I made the following tweak to what @milleruntime was doing, running 3 tservers on a single VM setup by uno.

apache/accumulo#2667 (comment)

It may not fail quickly, but leave it running and it will fail. I am seeing multiple different failures when I run this and leave it running for a while. I am running against the following branch of accumulo where I have added some more logging to compaction to see if I can track down some of the issues I am seeing.

https://github.com/keith-turner/accumulo/tree/compaction-logging

[DomGarguilo]

The code that prints this error message is here

accumulo-testing/src/main/java/org/apache/accumulo/testing/randomwalk/bulk/Verify.java

Line 73 in ddc6e58

throw new Exception("Bad key at " + entry);

Below that code is some code that analyzes markers, but it does not run when the count is not zero.

If this line was removed, do you think this exception would catch and print out the info you are looking for:

accumulo-testing/src/main/java/org/apache/accumulo/testing/randomwalk/bulk/Verify.java

Lines 93 to 95 in ddc6e58

	if (curr - 1 != prev)
	throw new Exception(
	"Bad marker count " + entry.getKey() + " " + entry.getValue() + " " + prev);

[DomGarguilo]

Also, are you able to reproduce the exception being thrown?

[Manno15]

If this line was removed, do you think this exception would catch and print out the info you are looking for:

I am currently looking into a solution that is basically that. I believe the other throw exception below will in theory always be thrown if this one is thrown because it compares the marker number.

I haven't been able to reproduce the exception on my end despite the suggestions above.

[keith-turner]

I haven't been able to reproduce the exception on my end despite the suggestions above.

The problem was intermittent. If you have the fix apache/accumulo@f6c943e then you may never see the bug.

For testing purposes, could create a bulk graph locally that will always cause verify to fail. I have not tested it, but maybe running the following graph would cause verify to always fail quickly. It does a plus one w/o a corresponding minus one and then verifies, so the counts should be non-zero.

<module>

<package prefix="bulk" value="org.apache.accumulo.testing.randomwalk.bulk"/>

<init id="bulk.Setup"/>

<node id="bulk.Setup">
  <edge id="bulk.BulkPlusOne" weight="1"/>
</node>

<node id="bulk.BulkPlusOne">
  <edge id="bulk.Verify" weight="10"/>
</node>

<node id="bulk.Verify">
  <edge id="END" weight="1"/>
</node>

</module>

[Manno15]

That is essentially what I did. Forced it to error at least. Was just commenting on seeing it naturally.

…

On Tue, Oct 18, 2022 at 6:33 AM Keith Turner ***@***.***> wrote: I haven't been able to reproduce the exception on my end despite the suggestions above. The problem was intermittent. If you have the fix ***@***.*** <apache/accumulo@f6c943e> then you may never see the bug. For testing purposes, could create a bulk graph locally that will always cause verify to fail. I have not tested it, but maybe running the following graph would cause verify to always fail quickly. It does a plus one w/o a corresponding minus one and then verifies, so the counts should be non-zero. <module> <package prefix="bulk" value="org.apache.accumulo.testing.randomwalk.bulk"/> <init id="bulk.Setup"/> <node id="bulk.Setup"> <edge id="bulk.BulkPlusOne" weight="1"/> </node> <node id="bulk.BulkPlusOne"> <edge id="bulk.Verify" weight="10"/> </node> <node id="bulk.Verify"> <edge id="END" weight="1"/> </node> </module> — Reply to this email directly, view it on GitHub <#239 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHASSV2AKYM5UQCFGNKGMKLWDZ4IHANCNFSM6AAAAAAQ7MBQQM> . You are receiving this because you commented.Message ID: ***@***.***>