Видео

Great explanation Ankur !!

Great video Ankur. Being following your content and blogs via linked-In Congrats !!

Hello I am getting error to read delta table that is on default at 01:21:50 IllegalArgumentException: Path must be absolute: default.customer_delta_table_persist.Please help me through that

Please mention the experience of the interviewee either in caption or in thumbnail. It would be helpful

The best playlist who wants to learn Kafka in data engineering domain. Every video has a clear cur explanation. Hope it will be continued. It can be a one stop tutorial for those who wants to learn kafka.

thanks!

The Python Program is an interesting one, assuming a value in list as Store ID and k as the distance or proximity within which another store with the same ID shouldn't exist. We can store a list as value with count,start_indx, end_ indx. If count is 2 we check the diff i.e., end_indx-start_indx <=k or not if it is then return that value and iterate the dict for the others.

dictionary = dict() for index, value in enumerate(nums): if value in dictionary: if abs(dictionary[value] - index) <= k: print(True) break dictionary[value] = index else: print(False)

Great series, can you upload more such videos with complex problems and bigger datasets

sum_val=0 def sum_of_vals_recursive(my_dict): global sum_val if (type(my_dict)==dict):#processing logic for dictionary for k,v in my_dict.items(): if(type(v)==int or type(v)==float):#if the value is a simple int/float sum_val=sum_val + v else: sum_of_vals_recursive(v)#this will get invoked during 1st nested level else:#this bit of logic is for nested ones, where it is list, set, tuple from 1st nesting for x in my_dict:#iteration logic for set, tuple, list if(type(x)==int or type(x)==float):#type check of elements of the value of each iterated from previous step sum_val = sum_val + x else: sum_of_vals_recursive(x)#this one is for handling next set of nestng and then, it will again follow return sum_val; inp_dict = {'ireland':100,'india':[200,300,[400,[200,200],400]],'uk':{'scotland':[50]}} print(sum_of_vals_recursive(inp_dict))

Sir in first pipeline I am getting error that str object has no attribute write

parquet store the file in hybrid format, not column based(80 % true)..

Is the next video available ?

Will i be able to switch into data engineering after watching and practicing the project ? Will i be able to tell my interview that i done this project in my current company?

can you please share the code ,dataset,api end point and also we are excepting one more session on optimal approach for this problem.

Can you share data Dataset links?

def distinct_ind(l,k): dict={} for i in range(len(l)): if l[i] in dict: if abs(i - dict[l[i]]) <=k: return True else: dict[l[i]]=i return False

Can you also setup data engineering interviews with respect to GCP ecosystem?

I think the first sql can be done using group by as well instead of window function

2.SQL solution: select name from ( select e.name, DATEDIFF(day,p.promotion_date,l.leave_start) as d_diff from employee e join promotions p on e.employee_id = p.employee_id join leaves l on e.employee_id = l.employee_id) A where d_diff = 1;

1.SQL solution: select name from ( select e.name,d.department_name, DATEDIFF(day,e.hire_date,p.promotion_date) as day_count, rank() over(partition by d.department_name order by DATEDIFF(day,e.hire_date,p.promotion_date) desc) as Rank from employee e join promotions p on e.employee_id =p.employee_id join departments d on e.department_id = d.department_id) A where rank =1;

great questions, thank you for this video

Awesome mock interview And the last question was very good How it saves time if you are reading from disk

it would be great if you attach the SQL and DSA questions in the comment or description

Please break down the video into topics.

WHAT AN EXPLANATION. KEEP UP THE GOOD WORK

Is this the full video? Where is the link? I thought it got abruptly cut at the end.

waiting for 70 days...

AMEX also asks the same question

Just completed this project after a lot of debugging. Got to learn about factory design pattern. Is this pattern typically used in the production environments? Thank you Ankur for creating such a quality project!

Where do we learn pyspark from scratch to advance with databricks

Hey Ankur This side Anshu, First of all, thanks for your amazing effort I'm a little bit confused about the source file (Extraction part) You explained to us in the videos We have used sources like CSV, Parquet, and Delta Table. But this is the type of file where you keep the data as a source then what is the Actual Source of data? For example, we have some ABC database I export the data in CSV or parquet and other file formats But my data source would be ABC Data Base) is it the right way I think? @Ankur

do really Interviewer give this much leverage, In most of the Interviews i gave, I was not even allowed to use any IDE, I have to write the code in Notepad. I'll suggest all the budding Data Engineers to practice and remember at least the basic syntax. window, to_date(), groupby() are very common and used extensively. \

is below correct ? df_count = example_df.count() ----> transformation example_df.count() ---> job ?