web-dev-qa-db-de.com

So verwenden Sie die NOT IN-Klausel im Filterzustand in funken

Ich möchte eine Spalte einer RDD-Quelle filtern:

val source = sql("SELECT * from sample.source").rdd.map(_.mkString(","))
val destination = sql("select * from sample.destination").rdd.map(_.mkString(","))

val source_primary_key = source.map(rec => (rec.split(",")(0)))
val destination_primary_key = destination.map(rec => (rec.split(",")(0)))

val src = source_primary_key.subtractByKey(destination_primary_key)

Ich möchte die IN-Klausel in der Filterbedingung verwenden, um nur die in src vorhandenen Werte aus der Quelle herauszufiltern, etwa unten (EDITED):

val source = spark.read.csv(inputPath + "/source").rdd.map(_.mkString(","))
val destination = spark.read.csv(inputPath + "/destination").rdd.map(_.mkString(","))

val source_primary_key = source.map(rec => (rec.split(",")(0)))
val destination_primary_key = destination.map(rec => (rec.split(",")(0)))

val extra_in_source = source_primary_key.filter(rec._1 != destination_primary_key._1)

äquivalenter SQL-Code ist 

SELECT * FROM SOURCE WHERE ID IN (select ID from src)

Vielen Dank

7
Vignesh

Da Ihr Code nicht reproduzierbar ist, finden Sie hier ein kleines Beispiel mit spark-sql zur Funktionsweise von select * from t where id in (...):

// create a DataFrame for a range 'id' from 1 to 9.
scala> val df = spark.range(1,10).toDF
df: org.Apache.spark.sql.DataFrame = [id: bigint]

// values to exclude
scala> val f = Seq(5,6,7)
f: Seq[Int] = List(5, 6, 7)

// select * from df where id is not in the values to exclude
scala> df.filter(!col("id").isin(f  : _*)).show
+---+                                                                           
| id|
+---+
|  1|
|  2|
|  3|
|  4|
|  8|
|  9|
+---+

// select * from df where id is in the values to exclude
scala> df.filter(col("id").isin(f  : _*)).show

Hier ist die RDD-Version von not isin:

scala> val rdd = sc.parallelize(1 to 10)
rdd: org.Apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[2] at parallelize at <console>:24

scala> val f = Seq(5,6,7)
f: Seq[Int] = List(5, 6, 7)

scala> val rdd2 = rdd.filter(x => !f.contains(x))
rdd2: org.Apache.spark.rdd.RDD[Int] = MapPartitionsRDD[3] at filter at <console>:28

Trotzdem glaube ich immer noch, dass dies ein Overkill ist, da Sie spark-sql bereits verwenden.

In Ihrem Fall handelt es sich anscheinend um DataFrames. Die oben genannten Lösungen funktionieren daher nicht.

Sie können den linken Anti-Join-Ansatz verwenden:

scala> val source = spark.read.format("csv").load("source.file")
source: org.Apache.spark.sql.DataFrame = [_c0: string, _c1: string ... 9 more fields]

scala> val destination = spark.read.format("csv").load("destination.file")
destination: org.Apache.spark.sql.DataFrame = [_c0: string, _c1: string ... 9 more fields]

scala> source.show
+---+------------------+--------+----------+---------------+---+---+----------+-----+---------+------------+
|_c0|               _c1|     _c2|       _c3|            _c4|_c5|_c6|       _c7|  _c8|      _c9|        _c10|
+---+------------------+--------+----------+---------------+---+---+----------+-----+---------+------------+
|  1|        Ravi kumar|   Ravi |     kumar|           MSO |  1|  M|17-01-1994| 74.5| 24000.78|    Alabama |
|  2|Shekhar shudhanshu| Shekhar|shudhanshu|      Manulife |  2|  M|18-01-1994|76.34|   250000|     Alaska |
|  3|Preethi Narasingam| Preethi|Narasingam|        Retail |  3|  F|19-01-1994|77.45|270000.01|    Arizona |
|  4|     Abhishek Nair|Abhishek|      Nair|       Banking |  4|  M|20-01-1994|78.65|   345000|   Arkansas |
|  5|        Ram Sharma|     Ram|    Sharma|Infrastructure |  5|  M|21-01-1994|79.12|    45000| California |
|  6|   Chandani Kumari|Chandani|    Kumari|          BNFS |  6|  F|22-01-1994|80.13| 43000.02|   Colorado |
|  7|      Balaji Kumar|  Balaji|     Kumar|           MSO |  1|  M|23-01-1994|81.33|  1234678|Connecticut |
|  8|  Naveen Shekrappa|  Naveen| Shekrappa|      Manulife |  2|  M|24-01-1994|  100|   789414|   Delaware |
|  9|     Milind Chavan|  Milind|    Chavan|        Retail |  3|  M|25-01-1994|83.66|   245555|    Florida |
| 10|      Raghu Rajeev|   Raghu|    Rajeev|       Banking |  4|  M|26-01-1994|87.65|   235468|     Georgia|
+---+------------------+--------+----------+---------------+---+---+----------+-----+---------+------------+


scala> destination.show
+---+-------------------+--------+----------+---------------+---+---+----------+-----+---------+------------+
|_c0|                _c1|     _c2|       _c3|            _c4|_c5|_c6|       _c7|  _c8|      _c9|        _c10|
+---+-------------------+--------+----------+---------------+---+---+----------+-----+---------+------------+
|  1|         Ravi kumar|   Revi |     kumar|           MSO |  1|  M|17-01-1994| 74.5| 24000.78|    Alabama |
|  1|        Ravi1 kumar|   Revi |     kumar|           MSO |  1|  M|17-01-1994| 74.5| 24000.78|    Alabama |
|  1|        Ravi2 kumar|   Revi |     kumar|           MSO |  1|  M|17-01-1994| 74.5| 24000.78|    Alabama |
|  2| Shekhar shudhanshu| Shekhar|shudhanshu|      Manulife |  2|  M|18-01-1994|76.34|   250000|     Alaska |
|  3|Preethi Narasingam1| Preethi|Narasingam|        Retail |  3|  F|19-01-1994|77.45|270000.01|    Arizona |
|  4|     Abhishek Nair1|Abhishek|      Nair|       Banking |  4|  M|20-01-1994|78.65|   345000|   Arkansas |
|  5|         Ram Sharma|     Ram|    Sharma|Infrastructure |  5|  M|21-01-1994|79.12|    45000| California |
|  6|    Chandani Kumari|Chandani|    Kumari|          BNFS |  6|  F|22-01-1994|80.13| 43000.02|   Colorado |
|  7|       Balaji Kumar|  Balaji|     Kumar|           MSO |  1|  M|23-01-1994|81.33|  1234678|Connecticut |
|  8|   Naveen Shekrappa|  Naveen| Shekrappa|      Manulife |  2|  M|24-01-1994|  100|   789414|   Delaware |
|  9|      Milind Chavan|  Milind|    Chavan|        Retail |  3|  M|25-01-1994|83.66|   245555|    Florida |
| 10|       Raghu Rajeev|   Raghu|    Rajeev|       Banking |  4|  M|26-01-1994|87.65|   235468|     Georgia|
+---+-------------------+--------+----------+---------------+---+---+----------+-----+---------+------------+

Sie müssen nur Folgendes tun:

scala> val res1 = source.join(destination, Seq("_c0"), "leftanti")

scala> val res2 = destination.join(source, Seq("_c0"), "leftanti")

Es ist dieselbe Logik, die ich in meiner Antwort hier erwähnte.

14
eliasah

Sie können versuchen wie--

df.filter(~df.Dept.isin("30","20")).show() 

// Dies listet alle Spalten von df auf, in denen sich NOT IN 30 oder 20 befindet

1
Blue Bird