Bulk import issue

[ciscodevnet]

Bulk import issue

[ciesinsn]

@pio-gra, I've tried to reproduce this and am unable. Do you have any other details to reproduce it maybe?

[joe-catanzaro]

I'm having a similar issue with bulk import. I get the CSV built according to the template, but it complains of illegal characters. Then I remove everything but the MAC addresses, and it states that all entries are filtered and "Successful Upload with exceptions on unformatted content". I don't believe I have any illegal characters in any of the fields, but it's hard to know when the output doesn't give any meaningful error handling.

Using iPSK manager on RHEL vm in Azure, uploading via Chrome browser.

[ciesinsn]

@joe-catanzaro just curious are you using the latest code that was committed a few weeks ago?

Is it possible for you to attach the file you are trying to upload that causes the issue so I can try and reproduce with that same file upload?

Thanks

Nick

[joe-catanzaro]

@joe-catanzaro just curious are you using the latest code that was committed a few weeks ago?

Is it possible for you to attach the file you are trying to upload that causes the issue so I can try and reproduce with that same file upload?

Thanks

Nick

hey Nick. this instance was just installed and brought running last week, I did run a git pull and it stated it's up to date.

I had to sanitize the file of the email address field, but that didn't seem to be the one that was causing problems. It seemed to be more the description field. Are there any character count limitations on that field? And I assume underscores are ok in the email field? I.e. [email protected]

problem file.csv

[joe-catanzaro]

@joe-catanzaro just curious are you using the latest code that was committed a few weeks ago?
Is it possible for you to attach the file you are trying to upload that causes the issue so I can try and reproduce with that same file upload?
Thanks
Nick

hey Nick. this instance was just installed and brought running last week, I did run a git pull and it stated it's up to date.

I had to sanitize the file of the email address field, but that didn't seem to be the one that was causing problems. It seemed to be more the description field. Are there any character count limitations on that field? And I assume underscores are ok in the email field? I.e. [email protected]

problem file.csv

@ciesinsn realized I forgot to tag, hopefully you saw this.

[ciesinsn]

@joe-catanzaro The file you attached I'm able to upload just fine, I assume it may be something in your description causing the issue. There is a filter that checks for invalid characters when a file is uploaded. When you upload the file do you see the count for invalid characters be anything other then zero? A code comment states " is a illegal character but I've found that & is also triggering the invalid character count and breaking a upload even though & is in the regular expression list. By chance are you using either of those in your file?

[joe-catanzaro]

So with no email addresses or descriptions, it still comes back and says ‘successful upload with exceptions on unformatted content. It recognizes the 209 entries, but shows all 209 entries filtered: ***@***.*** From: Nick Ciesinski ***@***.***> Sent: Saturday, November 11, 2023 4:45 PM To: CiscoDevNet/iPSK-Manager ***@***.***> Cc: Joe Catanzaro ***@***.***>; Mention ***@***.***> Subject: [EXT] Re: [CiscoDevNet/iPSK-Manager] Bulk import issue (Issue #23) ATTENTION: This email originated from outside of the organization. Please do not open any attachments or click any links unless you recognize the sender and know the content is safe. @joe-catanzaro<https://urldefense.com/v3/__https:/github.com/joe-catanzaro__;!!LVMHySo!iLfz4CULkgcLXWq-zmh422FoK0cqouxwE8-25E0Y1DdKB75Fh-8e12TNbnx93I-jrYQwlhUPL65a7Nj_WV-t_jqmm0-z$> The file you attached I'm able to upload just fine, I assume it may be something in your description causing the issue. There is a filter that checks for invalid characters when a file is uploaded. When you upload the file do you see the count for invalid characters be anything other then zero? A code comment states " is a illegal character but I've found that & is also triggering the invalid character count and breaking a upload even though & is in the regular expression list. By chance are you using either of those in your file? — Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https:/github.com/CiscoDevNet/iPSK-Manager/issues/23*issuecomment-1806924583__;Iw!!LVMHySo!iLfz4CULkgcLXWq-zmh422FoK0cqouxwE8-25E0Y1DdKB75Fh-8e12TNbnx93I-jrYQwlhUPL65a7Nj_WV-t_ol1hM6m$>, or unsubscribe<https://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/BDZS322XBPRZVOIPZUDTYGLYD7WVBAVCNFSM6AAAAAAQCESLA6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMBWHEZDINJYGM__;!!LVMHySo!iLfz4CULkgcLXWq-zmh422FoK0cqouxwE8-25E0Y1DdKB75Fh-8e12TNbnx93I-jrYQwlhUPL65a7Nj_WV-t_hd5cHv3$>. You are receiving this because you were mentioned.Message ID: ***@***.***> ====================================================================== The information in this email is confidential, and is intended solely for the addressee(s). Access to this email by anyone else is unauthorized and therefore prohibited. If you are not the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited and may be unlawful.

[ciesinsn]

@joe-catanzaro so this is when you try to upload the file you attached the other day? I have no issues with that file. Can you include a screenshot of the screen that shows whats been uploaded/filtered (the count). What version of PHP are you using and what OS is this on? Also, can you attach your php.ini file.

[joe-catanzaro]

@joe-catanzaro so this is when you try to upload the file you attached the other day? I have no issues with that file. Can you include a screenshot of the screen that shows whats been uploaded/filtered (the count). What version of PHP are you using and what OS is this on? Also, can you attach your php.ini file.

Yes, same file.

This is running on RHEL 9.2, PHP version info below:

PHP 8.0.30 (cli) (built: Aug 3 2023 17:13:08) ( NTS gcc x86_64 )
Copyright (c) The PHP Group
Zend Engine v4.0.30, Copyright (c) Zend Technologies
with Zend OPcache v8.0.30, Copyright (c), by Zend Technologies

PHP ini attached also
ipskmanager php ini.txt

[ciesinsn]

@joe-catanzaro, I'm on PHP 8.2 and our php.ini's are almost identical. Looking at your file as-is again (without my testing changes), I get the same issue. But after looking at the code I see why. Your file doesn't have a email listed and email is a required field.

iPSK-Manager/adminportal/ajax/fileupload.php

Lines 115 to 123 in 9ffede9

	if($tempMacAddress != "" && $tempFullname != "" && $tempEmail != ""){
	$csvFile[$entryCount]['macAddress'] = $tempMacAddress;
	$csvFile[$entryCount]['fullname'] = $tempFullname;
	$csvFile[$entryCount]['emailaddress'] = $tempEmail;
	$csvFile[$entryCount]['description'] = filter_var(trim($temp[$fieldMapping['description']]),FILTER_SANITIZE_STRING);
	$entryCount++;
	}else{
	$filtereditems++;
	}

Once you put a email in it should fix the filter issue (hopefully) and I think my pushed fixed for & will solve the description issue if that was a character you where using.

[joe-catanzaro]

Nick- When I had tried that previously, I get a different error. Our email addresses have underscores in them, is it possible that’s the issue? Error with emails added: [A screenshot of a computer Description automatically generated] From: Nick Ciesinski ***@***.***> Sent: Monday, November 13, 2023 9:14 PM To: CiscoDevNet/iPSK-Manager ***@***.***> Cc: Joe Catanzaro ***@***.***>; Mention ***@***.***> Subject: [EXT] Re: [CiscoDevNet/iPSK-Manager] Bulk import issue (Issue #23) ATTENTION: This email originated from outside of the organization. Please do not open any attachments or click any links unless you recognize the sender and know the content is safe. @joe-catanzaro<https://urldefense.com/v3/__https:/github.com/joe-catanzaro__;!!LVMHySo!nl8AFRN7892FU9b1U5tp0QET-Oo8hert9ZUdEfjPzFAKD7hYQMhRJGpLfEFfsqA6-dIu7mdRaskLsagH0Jam_byBi3yj$>, I'm on PHP 8.2 and our php.ini's are almost identical. Looking using your file as-is again (without my testing changes), I get the same issue. But after looking at the code I see why. Your file doesn't have a email listed and email is a required field. https://github.com/CiscoDevNet/iPSK-Manager/blob/9ffede91cd93a61aeaeb08deb532d1d931c12925/adminportal/ajax/fileupload.php#L115-L123<https://urldefense.com/v3/__https:/github.com/CiscoDevNet/iPSK-Manager/blob/9ffede91cd93a61aeaeb08deb532d1d931c12925/adminportal/ajax/fileupload.php*L115-L123__;Iw!!LVMHySo!nl8AFRN7892FU9b1U5tp0QET-Oo8hert9ZUdEfjPzFAKD7hYQMhRJGpLfEFfsqA6-dIu7mdRaskLsagH0Jam_ZT5I34U$> Once you put a email in it should fix the filter issue (hopefully) and I think my pushed fixed for & will solve the description issue if that was a character you where using. — Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https:/github.com/CiscoDevNet/iPSK-Manager/issues/23*issuecomment-1809433686__;Iw!!LVMHySo!nl8AFRN7892FU9b1U5tp0QET-Oo8hert9ZUdEfjPzFAKD7hYQMhRJGpLfEFfsqA6-dIu7mdRaskLsagH0Jam_c-zTjdC$>, or unsubscribe<https://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/BDZS32YVX34YFKGKEAG5RZLYELHX5AVCNFSM6AAAAAAQCESLA6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMBZGQZTGNRYGY__;!!LVMHySo!nl8AFRN7892FU9b1U5tp0QET-Oo8hert9ZUdEfjPzFAKD7hYQMhRJGpLfEFfsqA6-dIu7mdRaskLsagH0Jam_WOnzvnl$>. You are receiving this because you were mentioned.Message ID: ***@***.***> ====================================================================== The information in this email is confidential, and is intended solely for the addressee(s). Access to this email by anyone else is unauthorized and therefore prohibited. If you are not the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited and may be unlawful.

[joe-catanzaro]

@ciesinsn

[ciesinsn]

@joe-catanzaro, it was the underscore causing a issue. It was another character not being handled properly by the illegal character check along with &. I pushed a fix already and it does work for me. Give it a try and see if it fixes it for you as well.

[joe-catanzaro]

@ciesinsn so now I've done the git pull, restarted the box etc., and I can't get any file to upload to it now. Using chrome, i've updated the browser and disabled adblockers etc. but nothing happens when i click the upload button.

[ciesinsn]

@joe-catanzaro when was the last time you did a pull? The only things changed regarding this particular issue was two modifications to one line of code to handle a & and _ being in the CSV text. Is there any sort of error if you open developer tools on the browser?

[joe-catanzaro]

@ciesinsn did it again today just to be sure, return was 'already up to date'. In the dev tools I see this:

[ciesinsn]

Can you try and back rev that file only? That was the only one changed. Don't understand why you'd have a issue and I wouldn't, especially for the minor change that was done.

…

On Wed, Nov 15, 2023 at 1:33 PM joe-catanzaro ***@***.***> wrote: @ciesinsn <https://github.com/ciesinsn> did it again today just to be sure, return was 'already up to date'. In the dev tools I see this: [image: devtool error] <https://user-images.githubusercontent.com/150154731/283227721-63a08bc5-5b93-4b1c-b5ef-572c9d6f234f.PNG> — Reply to this email directly, view it on GitHub <#23 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AI5G4RIBBOCNXGWEZ5LW673YEUKKHAVCNFSM6AAAAAAQCESLA6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMJTGE2DANRXGQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[joe-catanzaro]

Can you try and back rev that file only? That was the only one changed. Don't understand why you'd have a issue and I wouldn't, especially for the minor change that was done.
…
On Wed, Nov 15, 2023 at 1:33 PM joe-catanzaro @.> wrote: @ciesinsn https://github.com/ciesinsn did it again today just to be sure, return was 'already up to date'. In the dev tools I see this: [image: devtool error] https://user-images.githubusercontent.com/150154731/283227721-63a08bc5-5b93-4b1c-b5ef-572c9d6f234f.PNG — Reply to this email directly, view it on GitHub <#23 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AI5G4RIBBOCNXGWEZ5LW673YEUKKHAVCNFSM6AAAAAAQCESLA6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMJTGE2DANRXGQ . You are receiving this because you were mentioned.Message ID: @.>

@ciesinsn sorry, how would I do that?

[ciesinsn]

https://github.com/CiscoDevNet/iPSK-Manager/tree/7db641c58a722e680fe16898b11b0b48bb59e683/adminportal/ajax

Pull file from that link and manually put it in place. @joe-catanzaro

[joe-catanzaro]

@ciesinsn ok, back rev'ed that file and noticed it was set for root as owner not apache. Fixed that, restarted HTTPD and still getting the same condition when I click 'upload' (nothing happens) but no more error in the dev tools console.

[ciesinsn]

Did you try a different csv just to make sure it wasn’t something with the file all the sudden? I’m not able to reproduce the issue at all. Before the last pull you did when was the last time you did a pull?

…

On Thu, Nov 16, 2023 at 8:16 AM joe-catanzaro ***@***.***> wrote: @ciesinsn <https://github.com/ciesinsn> ok, back rev'ed that file and noticed it was set for root as owner not apache. Fixed that, restarted HTTPD and still getting the same condition when I click 'upload' (nothing happens) but no more error in the dev tools console. — Reply to this email directly, view it on GitHub <#23 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AI5G4RLV5EMDXYZGU5ATXZDYEYN3RAVCNFSM6AAAAAAQCESLA6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMJUGUYTMOJWGY> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[joe-catanzaro]

I don't think I had done a pull since initial installation up until the update for this issue. I've used two different versions of that file, the 'problem file' is a copy of the actual one i've been working off of

[joe-catanzaro]

pulled a new template file, copied in some mac values and populated other values but same condition. it recognizes the file, lights up the upload button, but clicking 'upload' doesn't do anything and nothing happens in the dev console.

[ciesinsn]

@joe-catanzaro I can not reproduce any issue like that and I've tried a few files. Did you clear cache/history/etc.. in browser already?

Also, from within developer tools you should see in the network tab when you press upload it should place a new fileupload.php in the list, does it? if it does click it, go to the response tab, what is shown there, anything?

[joe-catanzaro]

Yes, I do see a new fileupload.php being generated each time. Output from response page attached.
Uploading response page.txt…

[ciesinsn]

@joe-catanzaro Looks like the file didn't upload right on your last message

[joe-catanzaro]

response page.txt

May not have waited long enough

[ciesinsn]

@joe-catanzaro Still think something isn't right with that file upload. There is a lot of text/code in it referencing github.

[joe-catanzaro]

I created a new CSV, manually typed in everything except I copy/paste values for the MAC addresses, nothing happens when I click upload.

[joe-catanzaro]

opened the CSV in notepad, nothing out of the ordinary in the file

[ciesinsn]

@joe-catanzaro I meant the upload of that response data from the web debugger

[joe-catanzaro]

is it maybe something to do with the fileupload.php file?

[ciesinsn]

@joe-catanzaro This is what I'm looking for from you from the developer tools in a browser which isn't coming through on your attachments in github correctly.

[joe-catanzaro]

Yes, all that text in the file I sent is what comes up in the 'response' tab

[ciesinsn]

@joe-catanzaro The text in the file you attached when I downloaded it has a lot of text that says "github". That in no way should be in that file.

[joe-catanzaro]

@joe-catanzaro The text in the file you attached when I downloaded it has a lot of text that says "github". That in no way should be in that file.

Something for sure went wrong pulling down that fileupload.php file. Going to try replacing it

[joe-catanzaro]

ok, fixed the fileupload.php file and now I'm at least getting responses, but back to square one with all of the csv's we've been working with here.

problem file still fails with Successful Upload with exceptions on unformatted content, illegal characters

original file with blank descriptions sees same failure as above

newly created file fails with Invalid CSV Header Attributes, even though they're identical to the template

[ciesinsn]

@joe-catanzaro Sure you have the latest version of the fileupload.php file? I've tried all your files and don't have any issues with the changes in the latest version. I even hand typed a few lines of your image a few messages above and don't have a issue.

[joe-catanzaro]

I've tried with both the previous version as well as the most recent one, doesn't seem to make a difference.

I think the more concerning thing is the file I hand typed out including headers and it's saying 'invalid header attributes'.

[ciesinsn]

@joe-catanzaro can you past in what line 105 looks like in your current fileupload.php file? Also, is there anything in the developer tools response other then just the one line of json with the counts of each issue?

[joe-catanzaro]

Line 105 reads:

$tempEntrydata = filter_var(trim($data),FILTER_VALIDATE_REGEXP, Array('options'=> Array('regexp' => "/^(?:([a-z]|[A-Z]|[0-9]|-|:|,|'|@|\.|;|\/|\(|\\|\&|#|\*|\s){1,})$/")));

Doesn't look like the underscore is in there.

Nothing other than the single line of JSON in the response

[ciesinsn]

@joe-catanzaro That is not the latest file then. It's also missing the & fix

[joe-catanzaro]

aha. that did it.

apparently what appeared to be the most recent file that I was looking at in github was not.

I went in and replaced line 105 with the updated version and now it worked. Thank you so much for your responsiveness and assistance with this!

[ciesinsn]

@joe-catanzaro no problem on the assistance. So just wanted to make sure, it sounds like all your uploads are working now, is that the case? if so I'll probably close out this mile long issue :)

[joe-catanzaro]

Yes, it appears that way. The ultimate resolution was the updated fileupload.php. Not sure why the header fields in the other file were no good, but it’s working now. From: Nick Ciesinski ***@***.***> Sent: Monday, November 20, 2023 12:12 PM To: CiscoDevNet/iPSK-Manager ***@***.***> Cc: Joe Catanzaro ***@***.***>; Mention ***@***.***> Subject: [EXT] Re: [CiscoDevNet/iPSK-Manager] Bulk import issue (Issue #23) ATTENTION: This email originated from outside of the organization. Please do not open any attachments or click any links unless you recognize the sender and know the content is safe. @joe-catanzaro<https://urldefense.com/v3/__https:/github.com/joe-catanzaro__;!!LVMHySo!laCYGsfDs48WkjZT4QqYXPlLSoFRA-vbez4Vz8Na3BZAAArcLPyqynaI5GrZ1zurKa8Kzl3NpAGWdG_h_jetK9w6AkLK$> no problem on the assistance. So just wanted to make sure, it sounds like all your uploads are working now, is that the case? if so I'll probably close out this mile long issue :) — Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https:/github.com/CiscoDevNet/iPSK-Manager/issues/23*issuecomment-1819482842__;Iw!!LVMHySo!laCYGsfDs48WkjZT4QqYXPlLSoFRA-vbez4Vz8Na3BZAAArcLPyqynaI5GrZ1zurKa8Kzl3NpAGWdG_h_jetK0PdLFdZ$>, or unsubscribe<https://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/BDZS3255FUOPFVFRITIDU6LYFOFNTAVCNFSM6AAAAAAQCESLA6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMJZGQ4DEOBUGI__;!!LVMHySo!laCYGsfDs48WkjZT4QqYXPlLSoFRA-vbez4Vz8Na3BZAAArcLPyqynaI5GrZ1zurKa8Kzl3NpAGWdG_h_jetKx-WfAcW$>. You are receiving this because you were mentioned.Message ID: ***@***.***> ====================================================================== The information in this email is confidential, and is intended solely for the addressee(s). Access to this email by anyone else is unauthorized and therefore prohibited. If you are not the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited and may be unlawful.