So verwenden Sie die NOT IN-Klausel im Filterzustand in funken

Question

Ich möchte eine Spalte einer RDD-Quelle filtern:

val source = sql("SELECT * from sample.source").rdd.map(_.mkString(",")) val destination = sql("select * from sample.destination").rdd.map(_.mkString(",")) val source_primary_key = source.map(rec => (rec.split(",")(0))) val destination_primary_key = destination.map(rec => (rec.split(",")(0))) val src = source_primary_key.subtractByKey(destination_primary_key)

Ich möchte die IN-Klausel in der Filterbedingung verwenden, um nur die in src vorhandenen Werte aus der Quelle herauszufiltern, etwa unten (EDITED):

val source = spark.read.csv(inputPath + "/source").rdd.map(_.mkString(",")) val destination = spark.read.csv(inputPath + "/destination").rdd.map(_.mkString(",")) val source_primary_key = source.map(rec => (rec.split(",")(0))) val destination_primary_key = destination.map(rec => (rec.split(",")(0))) val extra_in_source = source_primary_key.filter(rec._1 != destination_primary_key._1)

äquivalenter SQL-Code ist

SELECT * FROM SOURCE WHERE ID IN (select ID from src)

Vielen Dank

eliasah · Accepted Answer

Da Ihr Code nicht reproduzierbar ist, finden Sie hier ein kleines Beispiel mit spark-sql zur Funktionsweise von select * from t where id in (...):

// create a DataFrame for a range 'id' from 1 to 9. scala> val df = spark.range(1,10).toDF df: org.Apache.spark.sql.DataFrame = [id: bigint] // values to exclude scala> val f = Seq(5,6,7) f: Seq[Int] = List(5, 6, 7) // select * from df where id is not in the values to exclude scala> df.filter(!col("id").isin(f : _*)).show +---+ | id| +---+ | 1| | 2| | 3| | 4| | 8| | 9| +---+ // select * from df where id is in the values to exclude scala> df.filter(col("id").isin(f : _*)).show

Hier ist die RDD-Version von not isin:

scala> val rdd = sc.parallelize(1 to 10) rdd: org.Apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[2] at parallelize at <console>:24 scala> val f = Seq(5,6,7) f: Seq[Int] = List(5, 6, 7) scala> val rdd2 = rdd.filter(x => !f.contains(x)) rdd2: org.Apache.spark.rdd.RDD[Int] = MapPartitionsRDD[3] at filter at <console>:28

Trotzdem glaube ich immer noch, dass dies ein Overkill ist, da Sie spark-sql bereits verwenden.

In Ihrem Fall handelt es sich anscheinend um DataFrames. Die oben genannten Lösungen funktionieren daher nicht.

Sie können den linken Anti-Join-Ansatz verwenden:

scala> val source = spark.read.format("csv").load("source.file") source: org.Apache.spark.sql.DataFrame = [_c0: string, _c1: string ... 9 more fields] scala> val destination = spark.read.format("csv").load("destination.file") destination: org.Apache.spark.sql.DataFrame = [_c0: string, _c1: string ... 9 more fields] scala> source.show +---+------------------+--------+----------+---------------+---+---+----------+-----+---------+------------+ |_c0| _c1| _c2| _c3| _c4|_c5|_c6| _c7| _c8| _c9| _c10| +---+------------------+--------+----------+---------------+---+---+----------+-----+---------+------------+ | 1| Ravi kumar| Ravi | kumar| MSO | 1| M|17-01-1994| 74.5| 24000.78| Alabama | | 2|Shekhar shudhanshu| Shekhar|shudhanshu| Manulife | 2| M|18-01-1994|76.34| 250000| Alaska | | 3|Preethi Narasingam| Preethi|Narasingam| Retail | 3| F|19-01-1994|77.45|270000.01| Arizona | | 4| Abhishek Nair|Abhishek| Nair| Banking | 4| M|20-01-1994|78.65| 345000| Arkansas | | 5| Ram Sharma| Ram| Sharma|Infrastructure | 5| M|21-01-1994|79.12| 45000| California | | 6| Chandani Kumari|Chandani| Kumari| BNFS | 6| F|22-01-1994|80.13| 43000.02| Colorado | | 7| Balaji Kumar| Balaji| Kumar| MSO | 1| M|23-01-1994|81.33| 1234678|Connecticut | | 8| Naveen Shekrappa| Naveen| Shekrappa| Manulife | 2| M|24-01-1994| 100| 789414| Delaware | | 9| Milind Chavan| Milind| Chavan| Retail | 3| M|25-01-1994|83.66| 245555| Florida | | 10| Raghu Rajeev| Raghu| Rajeev| Banking | 4| M|26-01-1994|87.65| 235468| Georgia| +---+------------------+--------+----------+---------------+---+---+----------+-----+---------+------------+ scala> destination.show +---+-------------------+--------+----------+---------------+---+---+----------+-----+---------+------------+ |_c0| _c1| _c2| _c3| _c4|_c5|_c6| _c7| _c8| _c9| _c10| +---+-------------------+--------+----------+---------------+---+---+----------+-----+---------+------------+ | 1| Ravi kumar| Revi | kumar| MSO | 1| M|17-01-1994| 74.5| 24000.78| Alabama | | 1| Ravi1 kumar| Revi | kumar| MSO | 1| M|17-01-1994| 74.5| 24000.78| Alabama | | 1| Ravi2 kumar| Revi | kumar| MSO | 1| M|17-01-1994| 74.5| 24000.78| Alabama | | 2| Shekhar shudhanshu| Shekhar|shudhanshu| Manulife | 2| M|18-01-1994|76.34| 250000| Alaska | | 3|Preethi Narasingam1| Preethi|Narasingam| Retail | 3| F|19-01-1994|77.45|270000.01| Arizona | | 4| Abhishek Nair1|Abhishek| Nair| Banking | 4| M|20-01-1994|78.65| 345000| Arkansas | | 5| Ram Sharma| Ram| Sharma|Infrastructure | 5| M|21-01-1994|79.12| 45000| California | | 6| Chandani Kumari|Chandani| Kumari| BNFS | 6| F|22-01-1994|80.13| 43000.02| Colorado | | 7| Balaji Kumar| Balaji| Kumar| MSO | 1| M|23-01-1994|81.33| 1234678|Connecticut | | 8| Naveen Shekrappa| Naveen| Shekrappa| Manulife | 2| M|24-01-1994| 100| 789414| Delaware | | 9| Milind Chavan| Milind| Chavan| Retail | 3| M|25-01-1994|83.66| 245555| Florida | | 10| Raghu Rajeev| Raghu| Rajeev| Banking | 4| M|26-01-1994|87.65| 235468| Georgia| +---+-------------------+--------+----------+---------------+---+---+----------+-----+---------+------------+

Sie müssen nur Folgendes tun:

scala> val res1 = source.join(destination, Seq("_c0"), "leftanti") scala> val res2 = destination.join(source, Seq("_c0"), "leftanti")

Es ist dieselbe Logik, die ich in meiner Antwort hier erwähnte.

Blue Bird · Answer

Sie können versuchen wie--

df.filter(~df.Dept.isin("30","20")).show()

// Dies listet alle Spalten von df auf, in denen sich NOT IN 30 oder 20 befindet