If you're a massive Harry Potter fan it does make a difference to riding the HE from one location to the other, as it is a different experience going there and back, as what they show is different. A lot of fans feel it's essential since riding the HE is just an iconic moment in the books/films.
Taking that aside, if you're not a massive fan, it is a breeze traveling from one park to the other using the HE and other than waiting in the queue it's not only a breeze, but you're entertained in the process as well. If it's too long of a queue, you can always walk as well, which isn't that long of a walk. If anything I prefer to do so if I feel the desire to do something like Spider-Man over at IoA. As that attraction is closer to the entrance to the park. It is useful to be able to park hop if do if you notice a swift in crowds from one park to another, or if your interests change in wanting to do other things in another park.
And also going back to the HP stuff. If you're wanting to have the full HP experience, or simply have the luxury to wanting to do a lot of Potter at once / do both the same day, you'll have to get park to park.
However, it is not something that is necessary. Since you do have a lot of days you'll have plenty of time to do everything and taking your time to focus on each park.