dnif-archive/digivigi

digivigi's Issues

Scrapping only updated data from source website

Hey,
This is with regards to Process 2 - Stage 3.
After scrapping data from a source website, the next challenge was to fetch only that data which comes anew on a daily basis, as well as keep the old data.

I'm thinking of comparing date fields from source dataset and the system date to append only the new record to an existing csv. Hope it works out well.

META ISSUE: Additional Series of Issues Faced

Hello All,

@aakratisahu @shreyaskulkarni412 @Sharbanibasu23
This here is just a heads-up message which describes what - "META ISSUE: Additional Series of Issues Faced" is.

So, each of us have faced some minor/ individual issues which we did not think of as important at that time of occurrence but could be a hindrance for any person encountering them. Since our major purpose is to make this project as simple as possible, this sure does serve the purpose.

This is what we will be doing:

1. Every Issue will start with a naming convention. And the naming convention will be such;
   Issue#1:[Description of Issue in short]
2. These issues could be some which were raised on dnifHQ as well. But we'll have to frame them properly and provide links.

Have created an example issue which is a valid one, please check Issue#1: Log-In Problem onto DNIF Web Console

Post Script: This is just a description of what we will be doing. Do Not Close this Issue - It will be the last to be closed when the project ends.

DNIF installation on MAC

I have installed docker on mac machine. Then I execute "sudo docker-compose up" command. It shows me the ASCII of DNIF which is a sign of completion of the DNIF installation. But after that when I tried to connect with the source IP on [https://go.dnif.it/] . It is not connecting. But in case of Ubuntu machine, it works fine

Is there any specific issue for mac machine?

Need a web scrapper to capture tabular data

This is with reference to PROCESS 1 - Stage 3 given in README file.

Does anyone have a web scrapper that can scrape off data in tabular format on html pages and save it in a csv file.
Also need the header of this csv file to be custom edited such that the header names should start as '$headername' - this is a mandatory requirement.

Also please add comments to your code so its better to understand. Thanks. 👍

Configure SMTP to Send Alerts

Hey,
Could somebody configure SMTP inside DNIF container to send alerts from DNIF Web Console to our email Ids AND then document the exact detailed steps from Configuring TO "How To Send Alerts" since we're working on a similar environment?

@aakratisahu @Sharbanibasu23 Could you check this out? Since you're ahead in querying datasets comprehensively.
This could speed things up for us in the long run when we're nearing project completion.

Scheduling Repository Scripts

Hey Team Mates,

Does anyone know the right and easy way to schedule python scripts in Ubuntu, on a daily basis?

Need a quick one here.

Diagrammatic Overview of Project

Can anyone post a diagrammatic view of our project? It would help have a clear picture in mind and just so we are on the same page. I am working on one; will post it as soon as possible.

Capturing "How To?" of Process 1

Hi Guys,
Let us speed things up.
Currently Me and @Sharbanibasu23 are looking into the SMTP setup thing.

So maybe anyone of could volunteer to document the "How To?" of Process 1 of our project.
This will serve for future references.

We could have screenshots, brief informational steps, table of contents (for structuring this short process) Also we could use references in this "How To?" document of our other repository documents like installation docx, or analysis docx , or DNIF Website DOCS.

@shreyaskulkarni412 @aakratisahu ???

AWS log forwarding

Hello Team,

I have hosted one site on AWS which in node js and MongoDB. I want to forward the logs of the site to dnif.

Have anyone any idea regarding it??

Issue#2: Data not visible on DNIF Web Console

HTTP Listener service is ON.
IP and Port addresses are right.
API calling is also right.
But Data is not visible on DNIF Web Console.

installing DNIF on EC2 instance

I am trying to install the DNIF on EC2 instance of AWS for real-time streaming of our server logs. But get the problem in it. Is it due to less space in that instance.

Ubuntu 16.04 - Network Device Not Managed

Hi,
Please help me out with this issue. Stuck with this for a very long time now. It is stopping me from working any further.

Environment:
Windows -> VBox -> Ubuntu -> A Docker Container

Issue Arised When:
Did a requests.post(....) from my Ubuntu IP to it self [IP(A) -> IP(A)].
Soo... what happens is, my network manager says, device is not managed.
Cannot connect to internet. Cannot ping my self.

One time fix:
Changed - managed=false to true in NetworkManager.conf
But the it did not work out much.

Seems like my networking utilities are confused as to who is my DNS or its getting confused as to which network config to follow. (Although i did only one config. But the network manager did some overwriting for me or something when i did that post request)

Any solutions?
Anyone stumbled across this?

SMTP Configuration issue

I can not configure the SMTP on my machine is there specific steps for it

Dataset Hunting

Was just going through a set of data - mostly dynamic in nature and contextually cyber space inclined.
If there's more sources please add it to this discussion list. I'll soon create document featuring the list of probable sources we could work on in future.

Anyone is free to add to that doc by the way. Peace!

While configuring SMTP I am getting error as 'SMTP Configuration failed due to string indices must be integers, not str'.

Hi Everyone,
I am facing an issue while configuring SMTP in DNIF container. As @PRASHANT-SAWANT has already configured this I was following the steps mentioned by him #6. As mentioned in the steps, in smtp.yaml file I commented the internal settings and uncommented the external settings. I changed the external settings as follows-
domain: smtp.gmail.com
username: email id1
password: email id1 password
port: 587
tls: 1
from: email id1
to: email id2

After editing I saved the changes and ran the command $python configsmtp.py. But I am getting an error as 'SMTP Configuration failed due to string indices must be integers, not str'. I also opened the configsmtp.py file. But not being able to understand for which string object it is giving this error. If anyone of you have faced similar issue or have any solution for this please comment here.

Redundant code in repository

@Sharbanibasu23 @shreyaskulkarni412 @aakratisahu

I'm removing the redundant code from the repository.
I had originally kept PS_ConvertCsvToJson, PS_DailyDataFetch & PS_PostToAPI for the coding principle known as separation of concern.

But since the code is already merged and split into sections inside the script named - SourceToDnif, one could easily copy-paste it from there as required.

This removal was done to avoid further confusion and keep it simple.
P.S: Please pull the newer repository, locally.

starting Httplistner and httpslistner service

Hello,

For do certain things in dnif. I want to start http and https service. So anyone any idea for how to start this service?

Scraping through paged html tables

This is a good second issue, but was never posted here. But the solution is here for finding.
@Sharbanibasu23 Thanks. And really great job. I ran that script SB_WebScrapper_Using_Pandas_And_PageCrawling.py

I'd like to say its an excellent solution as well. We can further add to the code to send this data via POST request to DNIF Web API. And could fetch using the "event" data model.
This could be used for analysis of present and historical data for making predictions as well.

DNIF A10 Config File

Looking for a short description of DNIF A10 Components.

Issue#1: Log-In Problem onto DNIF Web Console

It happens sometimes that, when logging in to the DNIF Web Console, we are sometimes not able to do it in first time. After successive attempts it happens, but the number attempts is unknown - so we keep trying.

There seems to be a minor glitch of which we aren't aware of. Can someone enlighten?

@Sharbanibasu23 @aakratisahu @shreyaskulkarni412