- Practice from these notebooks throroughly and below pdf
- Incremental processing
- ETL patterns
- Data Privacy
- Performnace Optimization
- DLT practices
- Automate prod workflows
- Knowledge check
- Udemy Practice[I personally opted this which is MUST. Use Udemy for Bussiness for free]
Repo link
I was able to note down these topics memory based.
- Read parameters using dbutils.widgets.text and then get
- Read access for prod notebook for review to new data engineer - Can Read
- Attach notebook to cluster and run - Can Restart
- Production DLT pipelines in job cluster or all pupose cluster
- CTAS- executes load everytime or while table creation
- Scope access control to read prod secrets - Read on scope or secret
- %sh runs on driver node only
- If there is filter in query - file statistics in transaction log
- Vacuum run on shallow clone table- Error
- static_df.join(streaming_df) - which is not possible- left/inner/right
- Source is CDC - use merge into or leverage CDF feature
- How to find difference between previous and present commit
- Nighly jobs to overwrite a table for Bussiness team with least latency - Write to table nightly or create view
- What is Optimize Table - File target of 1GB?
- from code - .withWatermark for 10 inutes delay data
- from code - aggregate on source and overwrite/append to target
- Email notification in job run where mean(temp) > 120?. received email 3 time, why?
- Checkpoint should be unique for each streams
- Autoloader scenario based question in bronze with history and do update in target
- Streaming deduplication scenario from code question
- from code - batch load will overwrite/append?
- in CDF, if readChangeFeed start version is 0, and append, will there be deduplication?
- from code - Upsert- identify table is SCD 1 or 2
- To avoid performance issue, decrease trigger time or not
- delta table file skipping column decision based question - forst 32 questions, nested or not
- Grant usage and grand select on delta- what will it do?
- how to create unmanaged table?
- good candidate for partition column - date col
- Alter table xx rename xx - what happens in transaction log
- add check constraint error and recommendation
- tbl properties + comments + partition details - describe history/extended/detail?
- question on delta lake file statistics
- ganglia UI- detect spill based question
- repo branch missing on local - how to get that branch with latest code changes
- delete from A where id in (select id from B) - can we time travel and see those deleted records? and how to prevent?
- what is difference between dbfs and mounts
- api 2.0/jobs/create is exceuted 3 time using a json. what will happen? will it execute or create 3 jobs?
- What is dbfs
- install python lib - %pip install
- Task 1 has downstream task 2 and 3 in parallel. 1 passes, 2 passes, 3 fail- partially completed?
- streaming job retries in prod - job cluster, unlimited retries and 1 max concurrent run
- clone existing job and version it- how to do using databricks cli
- large json 1TB converted to parquet files with partition size of 512 MB - since its not delta table - read>narrow transformation>repartition 2048(1TB10241024/512)>convert to parquet? order of these steps?
- from code - drop duplicated on batch read and append- what happens in target table dedupliactes?
- one column was missed in prof from kafka- in future, to avoid - write to bronze to have full replayable history
- question on giving access control to users
- pyspark.sql.function.broadcast- what is the use- distribute to all worker nodes?
- from code - join on orders_is, when not matched- insert *- what it does
- def bronze_load is given. write silver load function so that it was be transformed and updated downstream
- case when is_member("group") then email else 'redacted end as email, lsv from table- what output if not member of group
- ganlia UI question to see logs based question
- how to get task having multi task run - 2.0/jobs/list or get ot 2.0/jobs/run/list or get
- what is unit testing
- in dev, multiple display() is executed repeatedly. what happens in prod?
- from delta, read has option("readChangeFeed") - will it work on source delta table with no CDC
- from code - identlty tumbling or sliding window
- Question on performance tuning spark.sql.files.maxPartitionBytes, spark.sql.shuffle.partitions
No matter what, please read these databricks docs. Note the Important tags in these pages and questions at the end on some pages.
- Data skipping with Z-order indexes for Delta Lake
- Clone a table on Databricks
- Delta table streaming reads and writes
- Structured Streaming Programming Guide - Spark 3.5.0 Documentation
- Configure Structured Streaming trigger intervals
- Configure Delta Lake to control data file size
- Introducing Stream-Stream Joins in Apache Spark 2.3
- Best Practices for Using Structured Streaming in Production - The Databricks Blog
- What is Auto Loader?
- Upsert into a Delta Lake table using merge
- Use Delta Lake change data feed on Databricks
- Apply watermarks to control data processing thresholds
- Use foreachBatch to write to arbitrary data sinks
- How to Simplify CDC With Delta Lake's Change Data Feed
- VACUUM
- Jobs access control
- Cluster access control
- Secret access control
- Hive metastore privileges and securable objects (legacy)
- Data objects in the Databricks lakehouse
- Constraints on Databricks
- When to partition tables on Databricks
- Manage clusters
- Export and import Databricks notebooks
- Unit testing for notebooks
- Databricks SQL Statement Execution API โ Announcing the Public Preview
- Transform data with Delta Live Tables
- Manage data quality with Delta Live Tables
- Simplified change data capture with the APPLY CHANGES API in Delta Live Tables
- Monitor Delta Live Tables pipelines
- Load data with Delta Live Tables
- What is Delta Live Tables?
- Solved: Re: What is the difference between Streaming live ... - Databricks - 17121
- What are all the Delta things in Databricks?
- Parameterized queries with PySpark
- Recover from Structured Streaming query failures with workflows
- Jobs API 2.0
- OPTIMIZE
- Adding and Deleting Partitions in Delta Lake tables
- What is the Databricks File System (DBFS)?
- Mounting cloud object storage on Databricks
- Databricks widgets
- Performance Tuning