koltyakov/cq-source-sharepoint

🔌 CloudQuery SharePoint Source Plugin

About me

I'm solution architect and developer with a focus on cloud. I live and work in Texas, USA. I'm a Microsoft MVP for M365 Development.

I don't like to keep experiense isolated, but prefer make an open source tool and bring it back to corp as a community proven. When you share you always get more in return.

It happens I know SharePoint a bit while dealing with the platform over a decade. But enterprise solutions are not limited to one platform, so there is a lot around: React, TypeScript, Node.js, Go, IaC, DevOps, Serverless, integrations, processes, orchestration.

GitHub profile stats

cq-source-sharepoint's Issues

Create data feed scripts and test with larger arrays of data

• Many various lists
• Large lists (10-50K something)

Sharepoint to Mysql : data randomly pushed to mysql table

Hi,

I have seen during my test phase that the rows are not pushed at every sync

I have attached the different scenarios

cloudquery - TableExistingNoRowsadded.log
cloudquery - TableExistingRowsAdded.log
cloudquery - TableCreatedNoRowsadded.log
cloudquery - TableCreatedRowsadded.log

Is it possible to use the sync to do a daily update ?

Thanks
Fred

Detect destination fields data types by entity response shape

E.g.

  lists:
     Shared Documents:
        select:
          - FileRef
          - Author/Title
          - File/Length
        expand:
          - Author
          - File

we can check that Author/Title is string and File/Length is int.

Add Managed Metadata support

Add support for user information list

Can this plugin get data when provided select sharepoint urls in format (doc_lib/folder_path) , instead of the entire list or doc lib of sharepoint

Can this plugin get data when provided select sharepoint urls in format (doc_lib/folder_path) , instead of the entire list or doc lib of sharepoint

Add User Profile Services support

Design naming transformation and namespaces

Tables in a destination should not conflict in advanced scenarios. Think out possible collisions with multi sites sync to the same destination.

feat: Migrate plugin to `github.com/cloudquery/plugin-sdk/v2`

Hello!
We've recently updated plugin-sdk to v2 (includes migration to Apache Arrow for the destination side).

Currently, no major updates to the source side were made.

Here's an easy step-by-step workflow that will allow you to migrate the code easily & efficiently:

1. Install gomajor
2. gomajor get github.com/cloudquery/plugin-sdk/v2
3. add replace for github.com/apache/arrow/go/v12 (see github.com/cloudquery/plugin-sdk/v2 go.mod)
4. go mod tidy && go get -t -d ./... && go mod tidy && go mod vendor
5. make gen lint test (if the appropriate targets exist)

Design data merge into a single destination table

It happens that some cases needs rollup of the different lists and/or lists from different sites into a single destination table.

Publish to CloudQuery Hub

Announcement of CloudQuery Hub points out to a new centralized way for publishing plugins.

As #22 is complete, packaging & publishing to Hub becomes available.

To verify the plugin can be packaged, go run main.go package --docs-dir docsdir -m 'test' v1.0.0 . should run OK.

Design question

Hey @koltyakov, Im starting to kick the tires on this one so I can open an initial version. A few Sharepoint and gosip related question. What would make sense for CloudQuery to sync, as far as I can understand sharepoint model and APIs, you need to specify what you want to select and what fields? is this correct? In that case I can make it a required configuration in addition to auth so the user can configure what they want to sync/fetch?

Thanks!
Yevgeny

Add content type based rollup

Embed advanced incremental sync and local state

Sync should support advanced increment behaviors shown in https://github.com/koltyakov/spsync. Technically spsync can be used for lists fetching.

csptl tool issue

Hi,

Thanks a lot for this Sharepoint Connector ! just doing some internal test sucessfully

Found 2 issues with csptl tool

• destination: should be renamed to destinations:
• tables: is now mandatory

Regards
Fred

Include mandatory fields into sync

Some fields should always be included no matter were they requested explicitly or not.

Such fields are:

• ID
• Created
• Modified
• Author
• Editor

Add search query results support

Is it possible to rename the fields?

@koltyakov, I wanted to rename the fields before saving to destination. Is there any way to achieve?

Shared Documents:
        select:
          - FileLeafRef
          - FileRef
          - FileDirRef
          - Author/Title
          - File/Length
          - EncodedAbsUrl

I wanted to rename EncodedAbsUrl to url and FileLeafRef to File_name

How to include file path?

@koltyakov, Thank you very much for your contribution to the plugin. I am testing the plugin to ingest data into Elasticsearch.
I wanted to include file path. Can you guide me on it?

lists:
    Shared Documents:
        select:
            - FileLeafRef
            - FileRef
            - FileDirRef
            - Author/Title
            - File/Length

Shared Documents File Details (SharePoint)

Elasticsearch Details

Add filter support for small entities

Using filters in large entities won't work, but it could be handy for a smaller lists.

Migrate plugin to `github.com/cloudqeury/plugin-sdk/v4`

UnauthorizedAccessException where connection worked previously

I have a very simple Sharepoint source connection that worked fine up until this morning. With no changes to the yaml, I am now getting:

Error: failed to sync v3 source sharepoint: rpc error: code = Internal desc = failed to init plugin: failed to initialize client: failed to connect to SharePoint: unable to request api: 403 Forbidden :: {"error":{"code":"-2147024891, System.UnauthorizedAccessException","message":{"lang":"en-US","value":"Attempted to perform an unauthorized operation."}}}

The same thing happens with 1.8.2 and 2.0.0.

spec:
  name: "sharepoint"
  registry: "github"
  path: "koltyakov/sharepoint"
  version: "v1.8.2"
  destinations: ["file"]
  tables: ["*"]
  spec:
    auth:
      strategy: "ondemand"
      creds:
        siteUrl: "https://sunpowercorp.sharepoint.com/sites/Engineering"
    lists:
      Lists/COE:

I have no problem hitting the list in a browser:
https://sunpowercorp.sharepoint.com/sites/Engineering/Lists/COE

Design plugin specification to correspond SharePoint specifics

• Authentication strategies params
• Source object query configuration